A Review of Explainable Deep Learning Cancer Detection Models in Medical Imaging
Source abstract
Abstract
Deep learning has demonstrated remarkable accuracy analyzing images for cancer detection tasks in recent years. The accuracy that has been achieved rivals radiologists and is suitable for implementation as a clinical tool. However, a significant problem is that these models are black-box algorithms therefore they are intrinsically unexplainable. This creates a barrier for clinical implementation due to lack of trust and transparency that is a characteristic of black box algorithms. Additionally, recent regulations prevent the implementation of unexplainable models in clinical settings which further demonstrates a need for explainability. To mitigate these concerns, there have been recent studies that attempt to overcome these issues by modifying deep learning architectures or providing after-the-fact explanations. A review of the deep learning explanation literature focused on cancer detection using MR images is presented here. The gap between what clinicians deem explainable and what current methods provide is discussed and future suggestions to close this gap are provided.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Explainable Artificial Intelligence (XAI) · Radiomics and Machine Learning in Medical Imaging · Artificial Intelligence in Healthcare and Education
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Science & Engineering Department, J.B. Speed School of Engineering, University of Louisville, Louisville, KT 40292, USA
References (100+)
Show source-provided references
1. Discriminating solitary cysts from soft tissue lesions in mammography using a pretrained deep convolutional neural network10.1002/mp.12110 ↗
2. Akselrod-Ballin, A., Karlinsky, L., Alpert, S., Hasoul, S., Ben-Ari, R., and Barkan, E. (2016). A Region Based Convolutional Network for Tumor Detection and Classification in Breast Mammography, in Deep Learning and Data Labeling for Medical Applications, Springer.10.1007/978-3-319-46976-8_21 ↗
3. Zhou, X., Kano, T., Koyasu, H., Li, S., Zhou, X., Hara, T., Matsuo, M., and Fujita, H. (2017). Automated assessment of breast tissue density in non-contrast 3D CT images without image segmentation based on a deep CNN. Medical Imaging 2017: Computer-Aided Diagnosis, International Society for Optics and Photonics.10.1117/12.2254320 ↗
4. SD-CNN: A shallow-deep CNN for improved breast cancer diagnosis10.1016/j.compmedimag.2018.09.004 ↗
5. Li, J., Fan, M., Zhang, J., and Li, L. (2017). Discriminating between benign and malignant breast tumors using 3D convolutional neural network in dynamic contrast enhanced-MR images. Medical Imaging 2017: Imaging Informatics for Healthcare, Research, and Applications, International Society for Optics and Photonics.10.1117/12.2254716 ↗
6. Deep Learning in Mammography: Diagnostic Accuracy of a Multipurpose Image Analysis Software in the Detection of Breast Cancer10.1097/RLI.0000000000000358 ↗
7. Jiang, F., Jiang, Y., Zhi, H., Dong, Y., Li, H., Ma, S., Wang, Y., Dong, Q., Shen, H., and Wang, Y. (2017). Artificial intelligence in healthcare past, present and future. Stroke Vasc. Neurol., 2.10.1136/svn-2017-000101 ↗
8. Tumor Detection in Automated Breast Ultrasound Using 3-D CNN and Prioritized Candidate Aggregation10.1109/TMI.2018.2860257 ↗
9. A Roadmap for Foundational Research on Artificial Intelligence in Medical Imaging10.1148/radiol.2019190613 ↗
10. Convolutional neural networks for computer-aided detection or diagnosis in medical image analysis: An overview10.3934/mbe.2019326 ↗
11. Large scale deep learning for computer aided detection of mammographic lesions10.1016/j.media.2016.07.007 ↗
12. Elsisi, M., Tran, M.-Q., Mahmoud, K., Lehtonen, M., and Darwish, M.M.F. (2021). Deep Learning-Based Industry 4.0 and Internet of Things towards Effective Energy Management for Smart Buildings. Sensors, 21.10.3390/s21041038 ↗
13. Zeiler, M.D., and Fergus, R. (2014, January 6–12). Visualizing and understanding convolutional networks. Proceedings of the 13th European Conference, Zurich, Switzerland. Lecture Notes in Computer Science.10.1007/978-3-319-10590-1_53 ↗
14. Zintgraf, L.M., Cohen, T.S., Adel, T., and Welling, M. (2017, January 24–26). Visualizing deep neural network decisions: Prediction difference analysis. Proceedings of the International Conference on Learning Representations, Toulon, France.
15. Elsisi, M., Mahmoud, K., Lehtonen, M., and Darwish, M.M.F. (2021). Reliable Industry 4.0 Based on Machine Learning and IoT for Analyzing, Monitoring, and Securing Smart Meters. Sensors, 21.10.3390/s21020487 ↗
16. Ali, M.N., Mahmoud, K., Lehtonen, M., and Darwish, M.M.F. (2021). Promising MPPT Methods Combining Metaheuristic, Fuzzy-Logic and ANN Techniques for Grid-Connected Photovoltaic. Sensors, 21.10.3390/s21041244 ↗
17. A novel deep learning based framework for the detection and classification of breast cancer using transfer learning. Pattern Recognition
18. Mohamad Ishak Desa, Lung cancer detection from CT image using improved profuse clustering and deep learning instantaneously trained neural networks10.1016/j.measurement.2019.05.027 ↗
19. Development and Validation of Deep Learning-based Automatic Detection Algorithm for Malignant Pulmonary Nodules on Chest Radiographs10.1148/radiol.2018180237 ↗
20. The European Union general data protection regulation: What it is and what it means10.1080/13600834.2019.1573501 ↗
21. Causability and explainability of artificial intelligence in medicine10.1002/widm.1312 ↗
22. Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D., and Gebru, T. (2018). Model cards for model reporting. arXiv.10.1145/3287560.3287596 ↗
23. Chen, R., Rodriguez, V., Grossman Liu, L.V., Mitchell, E.G., Averitt, A., Bear Don’t Walk, O., Bhave, S., Sun, T., and Thangaraj, P. (2018, January 16–18). Columbia DBMI CMS AI Challenge Team. Engendering Trust and Usability in Clinical Prediction of Unplanned Admissions: The CLinically Explainable Actionable Risk (CLEAR) Model. Proceedings of the Conference on Machine Learning for Health (MLHC), Stanford, CA, USA.
24. Zhang, X., Solar-Lezama, A., and Singh, R. (2018, January 3–8). Interpreting neural network judgments via minimal, stable, and symbolic corrections. Proceedings of the 32nd International Conference on Neural Information Processing Systems NIPS2018, Montreal, QC, Canada.
25. A Multisite Survey Study of EMR Review Habits, Information Needs, and Display Preferences among Medical ICU Clinicians Evaluating New Patients10.4338/ACI-2017-04-RA-0060 ↗
26. Ahern, I., Noack, A., Guzman-Nateras, L., Dou, D., Li, B., and Huan, J. (2019). NormLime: A New Feature Importance Metric for Explaining Deep Neural Networks. arXiv.
27. Hegselmann, S., Volkert, T., Ohlenburg, H., Gottschalk, A., Dugas, M., and Ertmer, C. (2020, January 7–8). An Evaluation of the Doctor-explainability of Generalized Additive Models with Interactions. Proceedings of the 5th Machine Learning for Healthcare Conference, PMLR, Vienna, Austria.
28. Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., and Brown, D. (2019). Text Classification Algorithms: A Survey. Information, 10.10.3390/info10040150 ↗
29. Lee, S., Lee, J., Lee, J., Park, C., and Yoon, S. (2018). Robust Tumor Localization with Pyramid Grad-CAM. arXiv.
30. An Improved Classification and Localization Approach to Small Bowel Capsule Endoscopy Using Convolutional Neural Network
31. Sumeet ShindeTanay ChouguleJitender SainiMadhura Ingalhalika (2019, January 13–17). HR-CAM: Precise Localization of Pathology Using Multi-Level Learning in CNNs. Medical Image Computing and Computer Assisted Intervention. Proceedings of the MICCAI 2019, 22nd International Conference, Shenzhen, China. Part IV.10.1007/978-3-030-32251-9_33 ↗
32. Gulum, M.A., Trombley, C.M., and Kantardzic, M. (September, January 31). Multiple Explanations Improve Deep Learning Transparency for Prostate Lesion Detection. Proceedings of the DMAH 2020, Waikoloa, HI, USA.10.1007/978-3-030-71055-2_11 ↗
33. The mythos of model interpretability10.1145/3233231 ↗
34. Tim, M. (2017). Explanation in Artificial Intelligence: Insights from the Social Sciences, arXiv e-prints, Computer Science—Artificial Intelligence. arXiv.
35. Narayanan, M., Chen, E., He, J., Kim, B., Gershman, S., and Doshi-Velez, F. (2018). How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-explainability of Explanation. arXiv.
36. Hicks, S., Riegler, M., Pogorelov, K., Anonsen, K.V., de Lange, T., Johansen, D., Jeppsson, M., Randel, K.R., Eskeland, S.L., and Halvorsen, P. (2018, January 18–21). Dissecting Deep Neural Networks for Better Medical Image Classification and Classification Understanding. Proceedings of the IEEE 31st International Symposium on Computer-Based Medical Systems (CBMS), Karlstad, Sweden.10.1109/CBMS.2018.00070 ↗
37. Opening the black box of neural networks: Methods for interpreting neural network models in clinical applications10.21037/atm.2018.05.32 ↗
38. Bhatt, U., Ravikumar, P., and Moura, J.M.F. (February, January 27). Building Human-Machine Trust via Interpretability. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
39. Sun, Y., Ravi, S., and Singh, V. (November, January 27). Adaptive Activation Thresholding: Dynamic Routing Type Behavior for Interpretability in Convolutional Neural Networks. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.10.1109/ICCV.2019.00504 ↗
40. Interpreting Deep Visual Representations via Network Dissection10.1109/TPAMI.2018.2858759 ↗
41. Jakab, T., Gupta, A., Bilen, H., and Vedaldi, A. (2020, January 16–18). Self-Supervised Learning of Interpretable Keypoints From Unlabelled Videos. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00881 ↗
42. Donglai, W., Bolei, Z., Antonio, T., and William, F. (2015). Understanding Intra-Class Knowledge Inside CNN. Computer Science—Computer Vision and Pattern Recognition. arXiv.
43. Chang, C., Creager, E., Goldenberg, A., and Duvenaud, D. (2019, January 6–9). Explaining Image Classifiers by Counterfactual Generation. Proceedings of the International Conference on Learning Representations ICLR, 2019, Ernest N. Morial Convention Center, New Orleans, LA, USA.
44. Yang, Y., and Song, L. (2020). Learn to Explain Efficiently via Neural Logic Inductive Learning. arXiv.
45. Oh, S.J., Augustin, M., Fritz, M., and Schiele, B. (May, January 30). Towards Reverse-Engineering Black-Box Neural Networks. Proceedings of the ICLR: 2018, Vancouver Convention Center, Vancouver, BC, Canada.10.1007/978-3-030-28954-6_7 ↗
46. Wang, T. (2019). Gaining Free or Low-Cost Transparency with explainable Partial Substitute. arXiv.
47. Kim, B., Wattenberg, M., Gilmer, J., Cai, C.J., Wexler, J., Viégas, F.B., and Sayres, R. (2018, January 10–15). Explainability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). Proceedings of the ICML, Stockholm, Sweden.
48. Chen, J., Song, L., Wainwright, M., and Jordan, M. (2018). Learning to Explain: An Information-Theoretic Perspective on Model explanation. arXiv.
49. Singla, S., Wallace, E., Feng, S., and Feizi, S. (2019). Understanding Impacts of High-Order Loss Approximations and Features in Deep Learning explanation. arXiv.
50. Schwab, P., and Karlen, W. (2019). CXPlain: Causal Explanations for Model explanation under Uncertainty. arXiv.
51. Guo, W., Huang, S., Tao, Y., Xing, X., and Lin, L. (2018). Explaining Deep Learning Models-A Bayesian Non-parametric Approach. arXiv.
52. Lage, I., Ross, A., Gershman, S.J., Kim, B., and Doshi-Velez, F. (2018). Human-in-the-Loop explainability Prior. arXiv.
53. Alvarez-Melis, D., and Jaakkola, T. (2018). Towards Robust explainability with Self-Explaining Neural Networks. arXiv.
54. Chen, C., Li, O., Barnett, A., Su, J., and Rudin, C. (2019, January 8–14). This Looks Like That: Deep Learning for Explainable Image Recognition. Proceedings of the NeurIPS: 2019, Vancouver Convention Center, Vancouver, BC, Canada.
55. Shrikumar, A., Greenside, P., and Kundaje, A. (2017). Learning Important Features Through Propagating Activation Differences. arXiv.
56. Feature combination networks for the explanation of statistical machine learning models: Application to Ames mutagenicity10.1186/1758-2946-6-8 ↗
57. Elish, M.C. (2018, January 7). The Stakes of Uncertainty: Developing and Integrating Machine Learning in Clinical Care. Proceedings of the 2018 EPIC Proceedings, Oxford, UK.10.1111/1559-8918.2018.01213 ↗
58. Ahmad, A.M., and Teredesai, E.C. (2018, January 4–7). Interpretable Machine Learning in Healthcare. Proceedings of the 2018 IEEE International Conference on Healthcare Informatics (ICHI), New York, NY, USA.10.1109/ICHI.2018.00095 ↗
59. RetainVis: Visual Analytics with explainable and Interactive Recurrent Neural Networks on Electronic Medical Records10.1109/TVCG.2018.2865027 ↗
60. Fong, R.C., and Vedaldi, A. (2017, January 27–29). Explainable explanations of black boxes by meaningful perturbation. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.371 ↗
61. Binder, A., Montavon, G., Lapuschkin, S., Muller, K., and Samek, W. (2016, January 6–9). Layer-Wise Relevance Propagation for Neural Networks with Local Renormalization Layers. Proceedings of the 25th International Conference on Artificial Neural Networks and Machine Learning (ICANN 2016), Barcelona, Spain.10.1007/978-3-319-44781-0_8 ↗
62. Explaining nonlinear classification decisions with deep Taylor decomposition10.1016/j.patcog.2016.11.008 ↗
63. Elshawi, R., Al-Mallah, M.H., and Sakr, S. (2019). On the explainability of machine learning-based model for predicting hypertension. BMC Med. Inform. Decis. Mak., 19.10.1186/s12911-019-0874-0 ↗
64. Gale, W., Oakden-Rayner, L., Gustavo, C., Andrew, P.B., and Lyle, J.P. (2018). Producing radiologist-quality reports for explainable artificial intelligence. arXiv.10.1109/ISBI.2019.8759236 ↗
65. Xie, Y., Chen, X.A., and Gao, G. (2019, January 20). Outlining the Design Space of Explainable Intelligent Systems for Medical Diagnosis. Proceedings of the ACM IUI 2019, Los Angeles, CA, USA.
66. Xie, P., Zuo, K., Zhang, Y., Li, F., Yin, M., and Lu, K. (2019). Interpretable Classification from Skin Cancer Histology Slides Using Deep Learning: A Retrospective Multicenter Study. arXiv.
67. Cruz-Roa, A., Arevalo, J., Madabhushi, A., and González, F. (2013, January 22–26). A Deep Learning Architecture for Image Representation, Visual explainability and Automated Basal-Cell Carcinoma Cancer Detection. Proceedings of the MICCAI International Conference on Medical Image Computing and Computer-Assisted Intervention, Nagoya, Japan.10.1007/978-3-642-40763-5_50 ↗
68. Zhang, R., Weber, C., Grossman, R., and Khan, A.A. (2020, January 7–8). Evaluating and interpreting caption prediction for histopathology images. Proceedings of the 5th Machine Learning for Healthcare Conference, in PMLR, Online Metting.
69. Hao, J., Kim, Y., Mallavarapu, T., Oh, J.H., and Kang, M. (2019). Explainable deep neural network for cancer survival analysis by integrating genomic and clinical data. BMC Med. Genom., 12.10.1186/s12920-019-0624-2 ↗
70. Al-Hussaini, I., Xiao, C., Westover, M.B., and Sun, J. (2019, January 9–10). SLEEPER: Interpretable Sleep staging via Prototypes from Expert Rules. Proceedings of the 4th Machine Learning for Healthcare Conference, in PMLR, Ann Arbor, MI, USA.
71. Essemlali, A., St-Onge, E., Descoteaux, M., and Jodoin, P. (2020, January 6–8). Understanding Alzheimer disease’s structural connectivity through explainable AI. Proceedings of the Third Conference on Medical Imaging with Deep Learning, PMLR, Montreal, QC, Canada.
72. The 2016 World Health Organization Classification of Tumors of the Central Nervous System: A summary10.1007/s00401-016-1545-1 ↗
73. Li, X., Cao, R., and Zhu, D. (2019). Vispi: Automatic Visual Perception and explanation of Chest X-rays. arXiv.
74. Grigorescu, I., Cordero-Grande, L., Edwards, A., Hajnal, J., Modat, M., and Deprez, M. (2019). Interpretable Convolutional Neural Networks for Preterm Birth Classification. arXiv.
75. Tjoa, E., and Guan, C. (2020). A Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI. IEEE Trans. Neural Netw. Learn. Syst.10.1109/TNNLS.2020.3027314 ↗
76. Singh, A., Sengupta, S., and Lakshminarayanan, V. (2020). Explainable Deep Learning Models in Medical Image Analysis. J. Imaging, 6.10.3390/jimaging6060052 ↗
77. Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI10.1016/j.inffus.2019.12.012 ↗
78. Singh, C., Murdoch, W.J., and Yu, B. (2019). Hierarchical explanations for neural network predictions. arXiv.
79. Adadi, A., and Berrada, M. (2018). Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI). IEEE Access.10.1109/ACCESS.2018.2870052 ↗
80. Lundberg, S.M., and Lee, S.I. (2017, January 4–9). A unified approach to interpreting model classifications. Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17), Long Beach, CA, USA.
81. de Sousa, I.P., Rebuzzi Vellasco, M.B., and da Silva, E.C. (2019). Local explainable Model-Agnostic Explanations for Classification of Lymph Node Metastases. Sensors, 19.10.3390/s19132969 ↗
82. Yang, C., Rangarajan, A., and Ranka, S. (2018, January 28–30). Global Model Interpretation Via Recursive Partitioning. Proceedings of the 2018 IEEE 20th International Conference on High Performance Computing and Communications; IEEE 16th International Conference on Smart City; IEEE 4th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), Exeter, UK.10.1109/HPCC/SmartCity/DSS.2018.00256 ↗
83. Interpreting neural network connection weights
84. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization10.1007/s11263-019-01228-7 ↗
85. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13–17). Why should i trust you? Explaining the classifications of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939778 ↗
86. Tonekaboni, S., Joshi, S., McCradden, M.D., and Goldenberg, A. (2020, January 7–8). What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use. Proceedings of the 4th Machine Learning for Healthcare Conference, Online Metting.
87. Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv.
88. Kim, B., Seo, J., Jeon, S., Koo, J., Choe, J., and Jeon, T. (November, January 27). Why are Saliency Maps Noisy? Cause of and Solution to Noisy Saliency Maps. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Korea.10.1109/ICCVW.2019.00510 ↗
89. Zhao, G., Zhou, B., Wang, K., Jiang, R., and Xu, M. (2018). Respond-CAM: Analyzing Deep Models for 3D Imaging Data by Visualizations, Medical Image Computing and Computer Assisted Intervention—MICCAI.10.1007/978-3-030-00928-1_55 ↗
90. Doshi-Velez, F., and Been, K. (2017). Towards A Rigorous Science of explainable Machine Learning. Machine Learning. arXiv.
91. Narayanan, B., Silva, M.S., Hardie, R., Kueterman, N.K., and Ali, R.A. (2019). Understanding Deep Neural Network Predictions for Medical Imaging Applications. arXiv.
92. Tomsett, R.J., Braines, D., Harborne, D., Preece, A., and Chakraborty, S. (2018). Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems. arXiv.
93. Chattopadhay, A., Sarkar, A., Howlader, P., and Balasubramanian, V.N. (2018, January 12–15). Grad-CAM++: Generalized Gradient-Based Visual Explanations for Deep Convolutional Networks. Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA.10.1109/WACV.2018.00097 ↗
94. Preece, A.D., Harborne, D., Braines, D., Tomsett, R., and Chakraborty, S. (2018). Stakeholders in Explainable AI. arXiv.
95. Hooker, S., Erhan, D., Kindermans, P., and Kim, B. (2019, January 8–14). A Benchmark for explainability Methods in Deep Neural Networks. Proceedings of the NeurIPS, Vancouver, BC, Canada.
96. Dementia in the elderly: An analysis of medical responsibility10.7326/0003-4819-94-6-802 ↗
97. Croskerry, P., Cosby, K., Graber, M.L., and Singh, H. (2017). Diagnosis: Interpreting the Shadows, CRC Press.10.1201/9781315116334 ↗
98. How far have we come? Artificial intelligence for chest radiograph interpretation10.1016/j.crad.2018.12.015 ↗
99. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (July, January 26). Learning Deep Features for Discriminative Localization. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.319 ↗
100. Windisch, P., Weber, P., Fürweger, C., Ehret, F., Kufeld, M., Zwahlen, D., and Muacevic, A. (2020). Implementation of model explainability for a basic brain tumor detection using convolutional neural networks on MRI slices. Neuroradiology.10.1007/s00234-020-02465-1 ↗