Multi-degree-of-freedom Intelligent Ultrasound Robot System Based on Reinforcement Learning

NING Guochen; ZHANG Xinran; LIAO Hongen

doi:10.11999/JEIT210879

Volume 44 Issue 1

Jan. 2022

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2022 > 44(1): 1-10

NING Guochen, ZHANG Xinran, LIAO Hongen. Multi-degree-of-freedom Intelligent Ultrasound Robot System Based on Reinforcement Learning[J]. Journal of Electronics & Information Technology, 2022, 44(1): 1-10. doi: 10.11999/JEIT210879

Citation:

NING Guochen, ZHANG Xinran, LIAO Hongen. Multi-degree-of-freedom Intelligent Ultrasound Robot System Based on Reinforcement Learning[J]. Journal of Electronics & Information Technology, 2022, 44(1): 1-10. doi: 10.11999/JEIT210879

Citation:

PDF( 4140 KB)

Multi-degree-of-freedom Intelligent Ultrasound Robot System Based on Reinforcement Learning

doi: 10.11999/JEIT210879

Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing 100084, China

Funds: The National Natural Science Foundation of China (82027807, 81771940), Beijing Municipal Natural Science Foundation (7212202), China Postdoctoral Science Foundation (2021M701928)

Received Date: 2021-08-26
Accepted Date: 2021-12-27
Rev Recd Date: 2021-12-23

Available Online: 2022-01-05

Publish Date: 2022-01-10

Abstract

Abstract

As a typical medical robot, the efficiency of ultrasound imaging and the fatigue caused by manual operation for a long time in assisted diagnosis and surgical guidance can effectively be reduced by ultrasound robots. To improve the imaging efficiency and stability of ultrasound robots in complex dynamic environments, a deep reinforcement learning-based imaging control method and system are proposed. Firstly, an imaging action decision method based on proximal policy gradient optimization is proposed to generate spatial action and probe pose motion decisions of the ultrasound robot in real-time and to realize the continuous generation process of imaging action decisions for targets in dynamic environments. Further, based on the characteristics of the complex and flexible environment faced by the ultrasound robot in the imaging task, an ultrasound robot control optimization strategy is proposed on the basis of the autonomous ultrasound robot motion decision. Eventually, a fully autonomous robotic ultrasound imaging process for different human body parts is achieved while avoiding parameter adjustments and complex dynamic environments.
- Ultrasound imaging,
- Ultrasound robotics,
- Soft control,
- Reinforcement learning

FullText(HTML)

References(21)

References

[1]	VON HAXTHAUSEN F, BÖTTGER F, WULFF S, et al. Medical robotics for ultrasound imaging: Current systems and future trends[J]. Current Robotics Reports, 2021, 2(1): 55–71. doi: 10.1007/s43154-020-00037-y
[2]	CHEN Fang, LIU Jia, and LIAO Hongen. 3D catheter shape determination for endovascular navigation using a two-step particle filter and ultrasound scanning[J]. IEEE Transactions on Medical Imaging, 2017, 36(3): 685–695. doi: 10.1109/TMI.2016.2635673
[3]	BEALES L, WOLSTENHULME S, EVANS J A, et al. Reproducibility of ultrasound measurement of the abdominal aorta[J]. British Journal of Surgery, 2011, 98(11): 1517–1525. doi: 10.1002/bjs.7628
[4]	CARDINAL H N, GILL J D, and FENSTER A. Analysis of geometrical distortion and statistical variance in length, area, and volume in a linearly scanned 3-D ultrasound image[J]. IEEE Transactions on Medical Imaging, 2000, 19(6): 632–651. doi: 10.1109/42.870670
[5]	SUNG G T and GILL I S. Robotic laparoscopic surgery: A comparison of the da Vinci and Zeus systems[J]. Urology, 2001, 58(6): 893–898. doi: 10.1016/S0090-4295(01)01423-6
[6]	HUANG Qinghua, LAN Jiulong, and LI Xuelong. Robotic arm based automatic ultrasound scanning for three-dimensional imaging[J]. IEEE Transactions on Industrial Informatics, 2019, 15(2): 1173–1182. doi: 10.1109/TII.2018.2871864
[7]	JIANG Zhongliang, ZHOU Yue, BI Yuan, et al. Deformation-aware robotic 3D ultrasound[J]. IEEE Robotics and Automation Letters, 2021, 6(4): 7675–7682. doi: 10.1109/LRA.2021.3099080
[8]	HOUSDEN J, WANG Shuangyi, BAO Xianqiang, et al. Towards standardized acquisition with a dual-probe ultrasound robot for fetal imaging[J]. IEEE Robotics and Automation Letters, 2021, 6(2): 1059–1065. doi: 10.1109/LRA.2021.3056033
[9]	JIANG Zhongliang, GRIMM M, ZHOU Mingchuan, et al. Automatic normal positioning of robotic ultrasound probe based only on confidence map optimization and force measurement[J]. IEEE Robotics and Automation Letters, 2020, 5(2): 1342–1349. doi: 10.1109/LRA.2020.2967682
[10]	MEROUCHE S, ALLARD L, MONTAGNON E, et al. A robotic ultrasound scanner for automatic vessel tracking and three-dimensional reconstruction of b-mode images[J]. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 2016, 63(1): 35–46. doi: 10.1109/TUFFC.2015.2499084
[11]	NING Guochen, ZHANG Xinran, and LIAO Hongen. Autonomic robotic ultrasound imaging system based on reinforcement learning[J] IEEE Transactions on Biomedical Engineering, 2021, 68(9): 2787–2797.
[12]	HENNERSPERGER C, FUERST B, VIRGA S, et al. Towards MRI-based autonomous robotic US acquisitions: A first feasibility study[J]. IEEE Transactions on Medical Imaging, 2017, 36(2): 538–548. doi: 10.1109/TMI.2016.2620723
[13]	CHATELAIN P, KRUPA A, and MARCHAL M. Real-time needle detection and tracking using a visually servoed 3D ultrasound probe[C]. 2013 IEEE International Conference on Robotics and Automation, Karlsruhe, Germany, 2013: 1676–1681.
[14]	LI Keyu, XU Yangxin, and MENG M Q H. An overview of systems and techniques for autonomous robotic ultrasound acquisitions[J]. IEEE Transactions on Medical Robotics and Bionics, 2021, 3(2): 510–524. doi: 10.1109/TMRB.2021.3072190
[15]	MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529–533. doi: 10.1038/nature14236
[16]	NGUYEN T T, NGUYEN N D, and NAHAVANDI S. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications[J]. IEEE Transactions on Cybernetics, 2020, 50(9): 3826–3839. doi: 10.1109/TCYB.2020.2977374
[17]	SUTTON R S and BARTO A G. Reinforcement Learning: An Introduction[M]. 2nd ed. Cambridge: MIT Press, 2018.
[18]	SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal policy optimization algorithms[J]. arXiv: 1707.06347, 2017.
[19]	SCHULMAN J, LEVINE S, ABBEEL P, et al. Trust region policy optimization[C]. The 32nd International Conference on Machine Learning, Lille, France, 2015: 1889–1897.
[20]	UGURLU B, HAVOUTIS I, SEMINI C, et al. Pattern generation and compliant feedback control for quadrupedal dynamic trot-walking locomotion: Experiments on RoboCat-1 and HyQ[J]. Autonomous Robots, 2015, 38(4): 415–437. doi: 10.1007/s10514-015-9422-7
[21]	VARIN P, GROSSMAN L, and KUINDERSMA S. A comparison of action spaces for learning manipulation tasks[C]. Proceedings of 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems, Macau, China, 2019: 6015–6021.