From Accuracy to Reliability and Robustness in Cardiac Magnetic Resonance Image Segmentation: A Review
Source abstract
Abstract
Since the rise of deep learning (DL) in the mid-2010s, cardiac magnetic resonance (CMR) image segmentation has achieved state-of-the-art performance. Despite achieving inter-observer variability in terms of different accuracy performance measures, visual inspections reveal errors in most segmentation results, indicating a lack of reliability and robustness of DL segmentation models, which can be critical if a model was to be deployed into clinical practice. In this work, we aim to bring attention to reliability and robustness, two unmet needs of cardiac image segmentation methods, which are hampering their translation into practice. To this end, we first study the performance accuracy evolution of CMR segmentation, illustrate the improvements brought by DL algorithms and highlight the symptoms of performance stagnation. Afterwards, we provide formal definitions of reliability and robustness. Based on the two definitions, we identify the factors that limit the reliability and robustness of state-of-the-art deep learning CMR segmentation techniques. Finally, we give an overview of the current set of works that focus on improving the reliability and robustness of CMR segmentation, and we categorize them into two families of methods: quality control methods and model improvement techniques. The first category corresponds to simpler strategies that only aim to flag situations where a model may be incurring poor reliability or robustness. The second one, instead, directly tackles the problem by bringing improvements into different aspects of the CMR segmentation model development process. We aim to bring the attention of more researchers towards these emerging trends regarding the development of reliable and robust CMR segmentation frameworks, which can guarantee the safe use of DL in clinical routines and studies.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced X-ray and CT Imaging · Medical Imaging Techniques and Applications · Advanced MRI Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Data Science Department, EURECOM, 06410 Biot, France
School of Biomedical Engineering and Imaging Sciences, King’s College London, London WC2R 2LS, UK
Funding
Agence Nationale de la Recherche
References (100+)
Show source-provided references
1. Global burden of cardiovascular diseases and risk factors, 1990–2019: Update from the GBD 2019 study10.1016/j.jacc.2020.11.010 ↗
2. World Health Organization (2022, February 01). Cardiovascular Diseases (CVDs) Fact Sheet. Available online: https://www.who.int/news-room/fact-sheets/detail/cardiovascular-diseases-(cvds).
3. Nelson, S., Whitsel, L., Khavjou, O., Phelps, D., and Leib, A. (2022, April 07). Projections of Cardiovascular Disease Prevalence and Costs. Available online: https://www.heart.org/-/media/Files/Get-Involved/Advocacy/CVD-Predictions-Through-2035.pdf.
4. World Health Organization (2013). Global Action Plan for the Prevention and Control of NCDs 2013–2020, WHO.
5. Quantification in cardiac MRI10.1002/(SICI)1522-2586(199911)10:5<602::AID-JMRI3>3.0.CO;2-C ↗
6. Automated cardiovascular magnetic resonance image analysis with fully convolutional networks10.1186/s12968-018-0471-x ↗
7. A review of segmentation methods in short axis cardiac MR images10.1016/j.media.2010.12.004 ↗
8. Challenges and methodologies of fully automatic whole heart segmentation: A review10.1260/2040-2295.4.3.371 ↗
9. Deep learning for cardiac image segmentation: A review10.3389/fcvm.2020.00025 ↗
10. A survey on deep learning in Medical Image Analysis10.1016/j.media.2017.07.005 ↗
11. Deep Learning Techniques for Automatic MRI Cardiac Multi-Structures Segmentation and Diagnosis: Is the Problem Solved?10.1109/TMI.2018.2837502 ↗
12. Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Elliott, P., Green, J., and Landray, M. (2015). UK Biobank: An Open Access Resource for Identifying the Causes of a Wide Range of Complex Diseases of Middle and Old Age. PLoS Med., 12.10.1371/journal.pmed.1001779 ↗
13. A collaborative resource to build consensus for automated left ventricular segmentation of cardiac MR images10.1016/j.media.2013.09.001 ↗
14. Right ventricle segmentation from cardiac MRI: A collation study10.1016/j.media.2014.10.004 ↗
15. Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Segmentation: The M&Ms Challenge10.1109/TMI.2021.3090082 ↗
16. Combining Registration and Minimum Surfaces for the Segmentation of the Left Ventricle in Cardiac Cine MR Images
17. An Exploration of 2D and 3D Deep Learning Techniques for Cardiac MR Image Segmentation
18. An Image-Based Comprehensive Approach for Automatic Segmentation of Left Ventricle from Cardiac Short Axis Cine MR Images10.1007/s10278-010-9315-4 ↗
19. Fast Fully-Automatic Cardiac Segmentation in MRI Using MRF Model Optimization, Substructures Tracking and B-Spline Smoothing
20. A dynamic elastic model for segmentation and tracking of the heart in MR image sequences10.1016/j.media.2010.05.009 ↗
21. Fully convolutional multi-scale residual DenseNets for cardiac segmentation and automated cardiac diagnosis using ensemble of classifiers10.1016/j.media.2018.10.004 ↗
22. DRAMMS: Deformable registration via attribute matching and mutual-saliency weighting10.1016/j.media.2010.07.002 ↗
23. Automatic Segmentation of LV and RV in Cardiac MRI
24. Layered Spatio-temporal Forests for Left Ventricle Segmentation from 4D Cardiac MRI Data
25. Automatic Cardiac Disease Assessment on cine-MRI via Time-Series Segmentation and Domain Specific Features
26. Automatic Segmentation of the Myocardium in Cine MR Images Using Deformable Registration
27. Class-Balanced Deep Neural Network for Automatic Ventricular Structure Segmentation
28. Automatic Left Ventricle Segmentation in Cardiac MRI Using Topological Stable-State Thresholding and Region Restricted Dynamic Programming10.1016/j.acra.2012.02.011 ↗
29. Quantitative CMR population imaging on 20,000 subjects of the UK Biobank imaging study: LV/RV quantification pipeline and its evaluation10.1016/j.media.2019.05.006 ↗
30. Wang, C.W., Peng, C.W., and Chen, H.C. (2012, January 1–5). A simple and fully automatic right ventricle segmentation method for 4-dimensional cardiac MR images. Proceedings of the MICCAI RV Segmentation Challenge, Nice, France.
31. AdaEn-Net: An ensemble of adaptive 2D-3D Fully Convolutional Networks for medical image segmentation10.1016/j.neunet.2020.03.007 ↗
32. Constantinides, C., Roullot, E., Lefort, M., and Frouin, F. (September, January 28). Fully automated segmentation of the left ventricle applied to cine MR images: Description and results on a database of 45 Subjects. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBC 2012, San Diego, CA, USA.10.1109/EMBC.2012.6346647 ↗
33. Domain-Adversarial Learning for Multi-Centre, Multi-Vendor, and Multi-Disease Cardiac MR Image Segmentation
34. Hybrid segmentation of left ventricle in cardiac MRI using gaussian-mixture model and region restricted dynamic programming10.1016/j.mri.2012.10.004 ↗
35. Disentangled Representations for Domain-Generalized Cardiac Segmentation
36. Multi-atlas Propagation Whole Heart Segmentation from MRI and CTA Using a Local Normalised Correlation Coefficient Criterion
37. Random Style Transfer Based Domain Generalization Networks Integrating Shape and Spatial Information
38. Ngo, T.A., and Carneiro, G. (2014, January 23–28). Fully Automated Non-rigid Segmentation with Distance Regularized Level Set Evolution Initialized and Constrained by Deep-Structured Inference. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.399 ↗
39. Style-Invariant Cardiac Image Segmentation with Test-Time Augmentation
40. Fast automatic myocardial segmentation in 4D cine CMR datasets10.1016/j.media.2014.06.001 ↗
41. Generalisable Cardiac Structure Segmentation via Attentional and Stacked Image Adaptation
42. Validation and development of a new automatic algorithm for time resolved segmentation of the left ventricle in magnetic resonance imaging10.1186/1532-429X-17-S1-P68 ↗
43. Cardiac MRI Segmentation With a Dilated CNN Incorporating Domain-Specific Constraints10.1109/JSTSP.2020.3013351 ↗
44. A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI10.1016/j.media.2016.01.005 ↗
45. Studying Robustness of Semantic Segmentation Under Domain Shift in Cardiac MRI
46. Tran, P.V. (2016). A Fully Convolutional Neural Network for Cardiac Segmentation in Short-Axis MRI. arXiv.
47. Histogram Matching Augmentation for Domain Adaptation with Application to Multi-centre, Multi-vendor and Multi-disease Cardiac Image Segmentation
48. Tan, L.K., Liew, Y.M., Lim, E., and McLaughlin, R.A. (2016, January 4–8). Cardiac left ventricle segmentation using convolutional neural network regression. Proceedings of the 2016 IEEE EMBS Conference on Biomedical Engineering and Sciences (IECBES), Kuala Lumpur, Malaysia.10.1109/IECBES.2016.7843499 ↗
49. Semi-supervised Cardiac Image Segmentation via Label Propagation and Style Transfer
50. 2D-3D Fully Convolutional Neural Networks for Cardiac MR Segmentation
51. A Deep Convolutional Neural Network Approach for the Segmentation of Cardiac Structures from MRI Sequences
52. Convolutional neural network regression for short-axis left ventricle segmentation in cardiac cine MR sequences10.1016/j.media.2017.04.002 ↗
53. Adaptive Preprocessing for Generalization in Cardiac MR Image Segmentation
54. Automatic Segmentation and Disease Classification Using Cardiac Cine MR Images
55. Multi-center, Multi-vendor, and Multi-disease Cardiac Image Segmentation Using Scale-Independent Multi-gate UNET
56. Automatic Multi-Atlas Segmentation of Myocardium with SVF-Net
57. A Generalizable Deep-Learning Approach for Cardiac Magnetic Resonance Image Segmentation Using Image Augmentation and Attention U-Net
58. GridNet with Automatic Shape Prior Registration for Automatic MRI Cardiac Segmentation
59. A 2-Step Deep Learning Method with Domain Adaptation for Multi-Centre, Multi-Vendor and Multi-Disease Cardiac Magnetic Resonance Segmentation
60. Densely Connected Fully Convolutional Network for Short-Axis Cardiac Cine MR Image Segmentation and Heart Diagnosis Using Random Forest
61. Deidentifying MRI Data Domain by Iterative Backpropagation
62. Deep Learning-Based Measurement of Total Plaque Area in B-Mode Ultrasound Images10.1109/JBHI.2021.3060163 ↗
63. Stability and generalization
64. Bahr, N.J. (2014). System Safety Engineering and Risk Assessment: A Practical Approach, CRC Press.
65. IEEE Standards Coordinating Committee (1990). IEEE Standard Glossary of Software Engineering Terminology. CA IEEE Comput. Soc., 169, 132.
66. Loss odyssey in medical image segmentation10.1016/j.media.2021.102035 ↗
67. Learning With Context Feedback Loop for Robust Medical Image Segmentation10.1109/TMI.2021.3060497 ↗
68. Sun, B., Feng, J., and Saenko, K. (2016, January 12–17). Return of frustratingly easy domain adaptation. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.10.1609/aaai.v30i1.10306 ↗
69. Gao, R., Liu, F., Zhang, J., Han, B., Liu, T., Niu, G., and Sugiyama, M. (2021, January 18–24). Maximum Mean Discrepancy Test is Aware of Adversarial Attacks. Proceedings of the 38th International Conference on Machine Learning, ICML 2021, Virtual Event.
70. Large-scale Quality Control of Cardiac Imaging in Population Studies: Application to UK Biobank10.1038/s41598-020-58212-2 ↗
71. Quantitative image quality evaluation of MR images using perceptual difference models10.1118/1.2903207 ↗
72. Lorch, B., Vaillant, G., Baumgartner, C., Bai, W., Rueckert, D., and Maier, A. (2017). Automated detection of motion artefacts in MR imaging using decision forests. J. Med. Eng., 2017.10.1155/2017/4501647 ↗
73. Semi-supervised Assessment of Incomplete LV Coverage in Cardiac MRI Using Generative Adversarial Nets10.1007/978-3-319-68127-6_7 ↗
74. Automatic CNN-based detection of cardiac MR motion artefacts using k-space data augmentation and curriculum learning10.1016/j.media.2019.04.009 ↗
75. Learning-Based Quality Control for Cardiac MR Images10.1109/TMI.2018.2878509 ↗
76. Machado, I., Puyol-Antón, E., Hammernik, K., Cruz, G., Ugurlu, D., Ruijsink, B., Castelo-Branco, M., Young, A., Prieto, C., and Schnabel, J.A. (2022). Quality-Aware Cine Cardiac MRI Reconstruction and Analysis from Undersampled K-Space Data. Statistical Atlases and Computational Models of the Heart. Multi-Disease, Multi-View, and Multi-Center Right Ventricular Segmentation in Cardiac MRI Challenge, Proceedings of the International Workshop on Statistical Atlases and Computational Models of the Heart, 12th International Workshop, STACOM 2021, Held in Conjunction with MICC, 12th International Workshop, Strasbourg, France, 27 September 2021, Springer.10.1007/978-3-030-93722-5_2 ↗
77. Fully Automated, Quality-Controlled Cardiac Analysis From CMR: Validation and Large-Scale Application to Characterize Cardiac Function10.1016/j.jcmg.2019.05.030 ↗
78. Automatic initialization and quality control of large-scale cardiac MRI segmentations10.1016/j.media.2017.10.001 ↗
79. Automated quantification of myocardial tissue characteristics from native T1 mapping using neural networks with uncertainty-based quality-control10.1186/s12968-020-00650-y ↗
80. Automatic segmentation with detection of local segmentation failures in cardiac MRI10.1038/s41598-020-77733-4 ↗
81. González, C., and Mukhopadhyay, A. (2021, January 7–9). Self-supervised Out-of-distribution Detection for Cardiac CMR Segmentation. Proceedings of the Medical Imaging with Deep Learning, Lübeck, Germany.
82. Evaluating Segmentation Error without Ground Truth
83. Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth10.1109/TMI.2017.2665165 ↗
84. Automated quality control in image segmentation: Application to the UK Biobank cardiovascular magnetic resonance imaging study10.1186/s12968-019-0523-x ↗
85. Real-Time Prediction of Segmentation Quality
86. Deep neural network ensemble for on-the-fly quality control-driven segmentation of cardiac MRI T1 mapping10.1016/j.media.2021.102029 ↗
87. Medical image segmentation automatic quality control: A multi-dimensional approach10.1016/j.media.2021.102213 ↗
88. Galati, F., and Zuluaga, M.A. (2021). Efficient Model Monitoring for Quality Control in Cardiac Image Segmentation. Functional Imaging and Modeling of the Heart, Proceedings of the International Conference on Functional Imaging and Modeling of the Heart, 11th International Conference, FIMH 2021, Stanford, CA, USA, 21–25 June 2021, Springer.10.1007/978-3-030-78710-3_11 ↗
89. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
90. Voxelwise atlas rating for computer assisted diagnosis: Application to congenital heart diseases of the great arteries10.1016/j.media.2015.09.001 ↗
91. Improving the generalizability of convolutional neural network-based segmentation on CMR images10.3389/fcvm.2020.00105 ↗
92. Improving cardiac MRI convolutional neural network segmentation on small training datasets and dataset shift: A continuous kernel cut approach10.1016/j.media.2020.101636 ↗
93. Convolutional Neural Network With Shape Prior Applied to Cardiac MRI Segmentation10.1109/JBHI.2018.2865450 ↗
94. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015, Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention 18th International Conference, Munich, Germany, 5–9 October 2015, Springer.10.1007/978-3-319-24574-4_28 ↗
95. Clough, J., Byrne, N., Oksuz, I., Zimmer, V.A., Schnabel, J.A., and King, A. (2020). A Topological Loss Function for Deep-Learning based Image Segmentation using Persistent Homology. arXiv.10.1007/978-3-030-20351-1_2 ↗
96. Wyburd, M.K., Dinsdale, N.K., Namburete, A.I.L., and Jenkinson, M. (2021). TEDS-Net: Enforcing Diffeomorphisms in Spatial Transformers to Guarantee Topology Preservation in Segmentations. Medical Image Computing and Computer Assisted Intervention-MICCAI 2021, Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, 24th International Conference, Strasbourg, France, 27 September–1 October 2021, Springer.10.1007/978-3-030-87193-2_24 ↗
97. Quality-Aware Semi-supervised Learning for CMR Segmentation
98. Cardiac Segmentation With Strong Anatomical Guarantees10.1109/TMI.2020.3003240 ↗
99. Galati, F., and Zuluaga, M.A. (2022). Using Out-of-Distribution Detection for Model Refinement in Cardiac Image Segmentation. Statistical Atlases and Computational Models of the Heart. Multi-Disease, Multi-View, and Multi-Center Right Ventricular Segmentation in Cardiac MRI Challenge, Proceedings of the International Workshop on Statistical Atlases and Computational Models of the Heart, 12th International Workshop, STACOM 2021, Held in Conjunction with MICC, 12th International Workshop, Strasbourg, France, 27 September 2021, Springer.10.1007/978-3-030-93722-5_40 ↗
100. Cardiac MR Segmentation from Undersampled k-space Using Deep Latent Representation Learning