A Coordinate-Regression-Based Deep Learning Model for Catheter Detection during Structural Heart Interventions
Source abstract
Abstract
With a growing geriatric population estimated to triple by 2050, minimally invasive procedures that are image-guided are becoming both more popular and necessary for treating a variety of diseases. To lower the learning curve for new procedures, it is necessary to develop better guidance systems and methods to analyze procedure performance. Since fluoroscopy remains the primary mode of visualizations, the ability to perform catheter tracking from fluoroscopic images is an important part of this endeavor. This paper explores the use of deep learning to perform the landmark detection of a catheter from fluoroscopic images in 3D-printed heart models. We show that a two-stage deep-convolutional-neural-network-based model architecture can provide improved performance by initially locating a region of interest before determining the coordinates of the catheter tip within the image. This model has an average error of less than 2% of the image resolution and can be performed within 4 milliseconds, allowing for its potential use for real-time intraprocedural tracking. Coordinate regression models have the advantage of directly outputting values that can be used for quantitative tracking in future applications and are easier to create ground truth values (~50× faster), as compared to semantic segmentation models that require entire masks to be made. Therefore, we believe this work has better long-term potential to be used for a broader class of cardiac devices, catheters, and guidewires.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Surgical Simulation and Training · Medical Image Segmentation Techniques · Advanced Neural Network Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Dalio Institute of Cardiovascular Imaging, NewYork-Presbyterian Hospital and Weill Cornell Medicine, New York, NY 10021, USA
Department of Radiology, Weill Cornell Medicine, New York, NY 10021, USA
AWS, Amazon, Seattle, WA 98170, USA
School of Electrical and Computer Engineering, Cornell University, Ithaca, NY 10021, USA
Division of Cardiology, Department of Medicine, Weill Cornell Medicine, New York, NY 10021, USA
References (75)
Show source-provided references
1. Minimally-invasive valve surgery10.1016/j.jacc.2010.03.053 ↗
2. The Utility of a 3D Endoscope and Robot-Assisted System for MIDCAB10.5761/atcs.oa.18-00254 ↗
3. Minimally invasive cardiac surgery: A systematic review and meta-analysis10.1016/j.ijcard.2016.08.227 ↗
4. Efficacy and Safety of Xinyue Capsule for Coronary Artery Disease after Percutaneous Coronary Intervention: A Systematic Review and Meta-Analysis of Randomized Clinical Trials
5. Global burden of CVD: Focus on secondary prevention of cardiovascular disease10.1016/S0167-5273(15)31026-3 ↗
6. Structural Heart Interventions10.14797/mdcj-13-3-96 ↗
7. Incidence and management of inadvertent puncture and sheath placement in the aorta during attempted transseptal puncture
8. Side-by-side comparison of fluoroscopy, 2D and 3D TEE during percutaneous edge-to-edge mitral valve repair10.1016/j.jcmg.2012.02.014 ↗
9. Novel system for real-time integration of 3-D echocardiography and fluoroscopy for image-guided cardiac interventions: Preclinical validation and clinical feasibility evaluation10.1109/JTEHM.2014.2303799 ↗
10. Identifying the third dimension in 2D fluoroscopy to create 3D cardiac maps10.1172/jci.insight.90453 ↗
11. Multimodality imaging for interventional cardiology10.2174/1381612823666170704171702 ↗
12. Hybrid imaging during transcatheter structural heart interventions10.1007/s12410-015-9349-6 ↗
13. Cardio navigation: Planning, simulation, and augmented reality in robotic assisted endoscopic bypass grafting10.1016/j.athoracsur.2004.11.060 ↗
14. Muraru, D., and Badano, L.P. (2017). Manual of 3D Echocardiography, Springer.
15. Development of a Hybrid Training Simulator for Structural Heart Disease Interventions10.1002/aisy.202000109 ↗
16. Liu, J., Al’Aref, S.J., Singh, G., Caprio, A., Moghadam, A.A.A., Jang, S.-J., Wong, S.C., Min, J.K., Dunham, S., and Mosadegh, B. (2019). An augmented reality system for image guidance of transcatheter procedures for structural heart disease. PLoS ONE, 14.10.1371/journal.pone.0219174 ↗
17. Deep learning-driven catheter tracking from bi-plane X-ray fluoroscopy of 3D printed heart phantoms
18. Use of extended realities in cardiology10.1016/j.tcm.2019.04.005 ↗
19. Virtual and Augmented Reality in Cardiovascular Care: State-of-the-Art and Future Perspectives10.1016/j.jcmg.2021.08.017 ↗
20. First-in-man experience with real-time holographic mixed reality display of three-dimensional echocardiography during structural intervention: Balloon mitral commissurotomy10.1093/eurheartj/ehz127 ↗
21. Extended, virtual and augmented reality in thoracic surgery: A systematic review10.1093/icvts/ivab241 ↗
22. Ronneberger, O., Fischer, P., and Brox, T. (2015, January 5–9). U-Net: Convolutional Networks for Biomedical Image Segmentation. Proceedings of the Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany.10.1007/978-3-319-24574-4_28 ↗
23. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. (2014, January 3–7). Caffe: Convolutional architecture for fast feature embedding. Proceedings of the 22nd ACM International Conference on Multimedia, Orlando, FL, USA.10.1145/2647868.2654889 ↗
24. Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., and LeCun, Y. (2013). Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv.
25. Chandan, G., Jain, A., and Jain, H. (2018, January 11–12). Real time object detection and tracking using Deep Learning and OpenCV. Proceedings of the 2018 International Conference on Inventive Research in Computing Applications (ICIRCA), Coimbatore, India.10.1109/ICIRCA.2018.8597266 ↗
26. Deepcut: Object segmentation from bounding box annotations using convolutional neural networks10.1109/TMI.2016.2621185 ↗
27. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
28. Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., and Ronneberger, O. (2016). International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.
29. Milletari, F., Navab, N., and Ahmadi, S.-A. (2016, January 25–28). V-net: Fully convolutional neural networks for volumetric medical image segmentation. Proceedings of the 2016 Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA.10.1109/3DV.2016.79 ↗
30. H-DenseUNet: Hybrid densely connected UNet for liver and tumor segmentation from CT volumes10.1109/TMI.2018.2845918 ↗
31. Zhou, Z., Rahman Siddiquee, M.M., Tajbakhsh, N., and Liang, J. (2018). Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer.
32. Zhang, J., Jin, Y., Xu, J., Xu, X., and Zhang, Y. (2018). Mdu-net: Multi-scale densely connected u-net for biomedical image segmentation. arXiv.
33. DUNet: A deformable network for retinal vessel segmentation10.1016/j.knosys.2019.04.025 ↗
34. RA-UNet: A hybrid deep attention-aware network to extract liver and tumor in CT scans10.3389/fbioe.2020.605132 ↗
35. Dolz, J., Ben Ayed, I., and Desrosiers, C. (2018). International MICCAI Brainlesion Workshop, Springer.
36. Guo, J., Deng, J., Xue, N., and Zafeiriou, S. (2018). Stacked dense u-nets with dual transformers for robust face alignment. arXiv.
37. Isensee, F., Petersen, J., Klein, A., Zimmerer, D., Jaeger, P.F., Kohl, S., Wasserthal, J., Koehler, G., Norajitra, T., and Wirkert, S. (2018). nnu-net: Self-adapting framework for u-net-based medical image segmentation. arXiv.10.1007/978-3-658-25326-4_7 ↗
38. Acute and sub-acute stroke lesion segmentation from multimodal MRI10.1016/j.cmpb.2020.105521 ↗
39. Dolz, J., Desrosiers, C., and Ben Ayed, I. (2018). International Workshop and Challenge on Computational Methods and Clinical Applications for Spine Imaging, Springer.
40. Zhuang, J. (2018). LadderNet: Multi-path networks based on U-Net for medical image segmentation. arXiv.
41. Oktay, O., Schlemper, J., Folgoc, L.L., Lee, M., Heinrich, M., Misawa, K., Mori, K., McDonagh, S., Hammerla, N.Y., and Kainz, B. (2018). Attention u-net: Learning where to look for the pancreas. arXiv.
42. Integrating geometric configuration and appearance information into a unified framework for anatomical landmark localization10.1016/j.media.2017.09.003 ↗
43. Landmark Detection in Cardiac MRI by Using a Convolutional Neural Network10.1148/ryai.2021200197 ↗
44. Detection and location of 127 anatomical landmarks in diverse CT datasets
45. Ibragimov, B., Likar, B., Pernus, F., and Vrtovec, T. (2016, January 13–16). Computerized Cephalometry by Game Theory with Shape-and Appearance-Based Landmark Refinement. Proceedings of the International Symposium on Biomedical imaging (ISBI), Prague, Czech Republic.
46. Automatic aorta segmentation and valve landmark detection in C-arm CT for transcatheter aortic valve implantation10.1109/TMI.2012.2216541 ↗
47. Stratified Decision Forests for Accurate Anatomical Landmark Localization in Cardiac Images10.1109/TMI.2016.2597270 ↗
48. Rohr, K. (2001). Landmark-Based Image Analysis using Geometric and Intensity Models, Springer.10.1007/978-94-015-9787-6 ↗
49. Zheng, Y.F., Liu, D., Georgescu, B., Nguyen, H., and Comaniciu, D. (2015, January 5–9). 3D Deep Learning for Efficient and Robust Landmark Detection in Volumetric Data. Proceedings of the Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015: 18th International Conference, Munich, Germany.10.1007/978-3-319-24553-9_69 ↗
50. Xu, Z., Huang, Q., Park, J., Chen, M., Xu, D., Yang, D., Liu, D., and Zhou, S.K. (2017, January 11–13). Supervised Action Classifier: Approaching Landmark Detection as Image Partitioning. Proceedings of the Medical Image Computing and Computer Assisted Intervention—MICCAI 2017: 20th International Conference, Quebec City, QC, Canada.10.1007/978-3-319-66179-7_39 ↗
51. Yang, D., Zhang, S.T., Yan, Z.N., Tan, C.W., Li, K., and Metaxas, D. (2015, January 16–19). Automated Anatomical Landmark Detection on Distal Femur Surface Using Convolutional Neural Network. Proceedings of the 2015 IEEE 12th International Symposium on Biomedical Imaging (Isbi), Brooklyn, NY, USA.10.1109/ISBI.2015.7163806 ↗
52. Nibali, A., He, Z., Morgan, S., and Prendergast, L. (2018). Numerical coordinate regression with convolutional neural networks. arXiv.
53. Toshev, A., and Szegedy, C. (2014, January 23–28). DeepPose: Human Pose Estimation via Deep Neural Networks. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.214 ↗
54. Chu, X., Yang, W., Ouyang, W.L., Ma, C., Yuille, A.L., and Wang, X.G. (2017, January 21–26). Multi-Context Attention for Human Pose Estimation. Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (Cvpr 2017), Honolulu, HI, USA.10.1109/CVPR.2017.601 ↗
55. Li, J., Wang, Y., Mao, J., Li, G., and Ma, R. (2020). End-to-End Coordinate Regression Model with Attention-Guided Mechanism for Landmark Localization in 3D Medical Images. International Workshop on Machine Learning in Medical Imaging, Springer.10.1007/978-3-030-59861-7_63 ↗
56. Dünnwald, M., Betts, M.J., Düzel, E., and Oeltze-Jafra, S. (2021). Bildverarbeitung für die Medizin 2021, Springer.
57. Pixel-in-pixel net: Towards efficient facial landmark detection in the wild10.1007/s11263-021-01521-4 ↗
58. A survey of catheter tracking concepts and methodologies10.1016/j.media.2022.102584 ↗
59. Guidewire tracking during endovascular neurosurgery10.1016/j.medengphy.2010.05.006 ↗
60. Robust guidewire tracking under large deformations combining segment-like features (SEGlets)10.1016/j.media.2017.02.001 ↗
61. Wang, P., Chen, T., Zhu, Y., Zhang, W., Zhou, S.K., and Comaniciu, D. (2009, January 20–25). Robust guidewire tracking in fluoroscopy. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA.10.1109/CVPR.2009.5206692 ↗
62. Zweng, M., Fallavollita, P., Demirci, S., Kowarschik, M., Navab, N., and Mateus, D. (2015, January 6). Automatic guide-wire detection for neurointerventions using low-rank sparse matrix decomposition and denoising. Proceedings of the Workshop on Augmented Environments for Computer-Assisted Interventions, Munich, Germany.10.1007/978-3-319-24601-7_12 ↗
63. Ambrosini, P., Ruijters, D., Niessen, W.J., Moelker, A., and Walsum, T.v. (2017, January 11–13). Fully automatic and real-time catheter segmentation in X-ray fluoroscopy. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Quebec City, QC, Canada.10.1007/978-3-319-66185-8_65 ↗
64. Nguyen, A., Kundrat, D., Dagnino, G., Chi, W., Abdelaziz, M.E., Guo, Y., Ma, Y., Kwok, T.M., Riga, C., and Yang, G.-Z. (2020, January 1). End-to-end real-time catheter segmentation with optical flow-guided warping during endovascular intervention. Proceedings of the 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France.10.1109/ICRA40945.2020.9197307 ↗
65. Subramanian, V., Wang, H., Wu, J.T., Wong, K.C., Sharma, A., and Syeda-Mahmood, T. (2019, January 13–17). Automated detection and type classification of central venous catheters in chest X-rays. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Shenzhen, China.10.1007/978-3-030-32226-7_58 ↗
66. A real-time multifunctional framework for guidewire morphological and positional analysis in interventional X-ray fluoroscopy10.1109/TCDS.2020.3023952 ↗
67. Li, R.-Q., Bian, G., Zhou, X., Xie, X., Ni, Z., and Hou, Z. (2019, January 13–17). A two-stage framework for real-time guidewire endpoint localization. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Shenzhen, China.10.1007/978-3-030-32254-0_40 ↗
68. Vlontzos, A., and Mikolajczyk, K. (2018). Deep segmentation and registration in X-ray angiography video. arXiv.
69. Deep learning-based framework for motion-compensated image fusion in catheterization procedures10.1016/j.compmedimag.2022.102069 ↗
70. The challenges facing deep learning–based catheter localization for ultrasound guided high-dose-rate prostate brachytherapy10.1002/mp.15522 ↗
71. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the 3rd International Conference on Learning Representations (ICLR2015), San Diego, CA, USA.
72. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., and Fei-Fei, L. (2009, January 20–25). Imagenet: A large-scale hierarchical image database. Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR2009), Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
73. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv.
74. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
75. Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K.Q. (2017, January 21–26). Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.243 ↗