Recent Applications of Explainable AI (XAI): A Systematic Literature Review
Source abstract
Abstract
This systematic literature review employs the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology to investigate recent applications of explainable AI (XAI) over the past three years. From an initial pool of 664 articles identified through the Web of Science database, 512 peer-reviewed journal articles met the inclusion criteria—namely, being recent, high-quality XAI application articles published in English—and were analyzed in detail. Both qualitative and quantitative statistical techniques were used to analyze the identified articles: qualitatively by summarizing the characteristics of the included studies based on predefined codes, and quantitatively through statistical analysis of the data. These articles were categorized according to their application domains, techniques, and evaluation methods. Health-related applications were particularly prevalent, with a strong focus on cancer diagnosis, COVID-19 management, and medical imaging. Other significant areas of application included environmental and agricultural management, industrial optimization, cybersecurity, finance, transportation, and entertainment. Additionally, emerging applications in law, education, and social care highlight XAI’s expanding impact. The review reveals a predominant use of local explanation methods, particularly SHAP and LIME, with SHAP being favored for its stability and mathematical guarantees. However, a critical gap in the evaluation of XAI results is identified, as most studies rely on anecdotal evidence or expert opinion rather than robust quantitative metrics. This underscores the urgent need for standardized evaluation frameworks to ensure the reliability and effectiveness of XAI applications. Future research should focus on developing comprehensive evaluation standards and improving the interpretability and stability of explanations. These advancements are essential for addressing the diverse demands of various application domains while ensuring trust and transparency in AI systems.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Explainable Artificial Intelligence (XAI) · Machine Learning in Healthcare · Artificial Intelligence in Healthcare and Education
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Information Technology, University of Jyväskylä, P.O. Box 35, FI-40014 Jyväskylä, Finland
Faculty of Electrical Engineering and Computer Science, University of Maribor, 2000 Maribor, Slovenia
Funding
K.H. Renlund Foundation and the Academy of Finland · Slovenian Research Agency
References (100+)
Show source-provided references
1. Peeking inside the black-box: A survey on explainable artificial intelligence (XAI)10.1109/ACCESS.2018.2870052 ↗
2. Explainable artificial intelligence: A comprehensive review10.1007/s10462-021-10088-y ↗
3. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities10.1016/j.knosys.2023.110273 ↗
4. From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai10.1145/3583558 ↗
5. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI10.1016/j.inffus.2019.12.012 ↗
6. Hu, Z.F., Kuflik, T., Mocanu, I.G., Najafian, S., and Shulner Tal, A. (2021, January 21–25). Recent studies of xai-review. Proceedings of the Adjunct 29th ACM Conference on User Modeling, Adaptation and Personalization, Utrecht, The Netherlands.10.1145/3450614.3463354 ↗
7. Islam, M.R., Ahmed, M.U., Barua, S., and Begum, S. (2022). A systematic review of explainable artificial intelligence in terms of different application domains and tasks. Appl. Sci., 12.10.3390/app12031353 ↗
8. A systematic review of Explainable Artificial Intelligence models and applications: Recent developments and future trends10.1016/j.dajour.2023.100230 ↗
9. A comprehensive taxonomy for explainable artificial intelligence: A systematic survey of surveys on methods and concepts10.1007/s10618-022-00867-8 ↗
10. Speith, T. (2022, January 21–24). A review of taxonomies of explainable artificial intelligence (XAI) methods. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea.10.1145/3531146.3534639 ↗
11. Notions of explainability and evaluation approaches for explainable artificial intelligence10.1016/j.inffus.2021.05.009 ↗
12. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement10.7326/0003-4819-151-4-200908180-00135 ↗
13. Samek, W., Montavon, G., Vedaldi, A., Hansen, L.K., and Müller, K.R. (2019). Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer Nature.10.1007/978-3-030-28954-6 ↗
14. Koh, P.W., and Liang, P. (2017, January 6–11). Understanding black-box predictions via influence functions. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia.
15. Yeh, C.K., Kim, J., Yen, I.E.H., and Ravikumar, P.K. (2018). Representer point selection for explaining deep neural networks. Adv. Neural Inf. Process. Syst., 31.
16. Li, O., Liu, H., Chen, C., and Rudin, C. (2018, January 2–7). Deep Learning for Case-Based Reasoning through Prototypes: A Neural Network that Explains Its Predictions. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, Orleans, LA, USA.10.1609/aaai.v32i1.11771 ↗
17. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR
18. Erhan, D., Bengio, Y., Courville, A., and Vincent, P. (2009). Visualizing higher-layer features of a deep network. Univ. Montr., 1341.
19. Extracting refined rules from knowledge-based neural networks10.1007/BF00993103 ↗
20. Interpretation of artificial neural networks by means of fuzzy rules10.1109/72.977279 ↗
21. Neuro-fuzzy rule generation: Survey in soft computing framework10.1109/72.846746 ↗
22. All Models are Wrong, but Many are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously
23. Fong, R.C., and Vedaldi, A. (2017, January 22–29). Interpretable explanations of black boxes by meaningful perturbation. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.371 ↗
24. Zintgraf, L.M., Cohen, T.S., Adel, T., and Welling, M. (2017, January 24–26). Visualizing deep neural network decisions: Prediction difference analysis. Proceedings of the International Conference on Learning Representations, ICLR, Toulon, France.
25. Zeiler, M.D., and Fergus, R. (2014, January 6–12). Visualizing and understanding convolutional networks. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10590-1_53 ↗
26. Comparison of feature importance measures as explanations for classification models10.1007/s42452-021-04148-9 ↗
27. Wojtas, M., and Chen, K. (2020, January 6–12). Feature Importance Ranking for Deep Learning. Proceedings of the Advances in Neural Information Processing Systems (NIPS 2020), Vancouver, BC, Canada.
28. A Survey on the Explainability of Supervised Machine Learning10.1613/jair.1.12228 ↗
29. Saarela, M. (2024, January 8–12). On the relation of causality-versus correlation-based feature selection on model fairness. Proceedings of the 39th ACM/SIGAPP Symposium on Applied Computing, Avila, Spain.10.1145/3605098.3636018 ↗
30. A survey of methods for explaining black box models
31. Molnar, C. (2020). Interpretable Machine Learning, Lulu. com.
32. Saarela, M., and Geogieva, L. (2022). Robustness, Stability, and Fidelity of Explanations for a Deep Skin Cancer Classification Model. Appl. Sci., 12.10.3390/app12199545 ↗
33. Carvalho, D.V., Pereira, E.M., and Cardoso, J.S. (2019). Machine learning interpretability: A survey on methods and metrics. Electronics, 8.10.3390/electronics8080832 ↗
34. Wang, Y., Zhang, T., Guo, X., and Shen, Z. (2024). Gradient based Feature Attribution in Explainable AI: A Technical Review. arXiv.
35. Can we automate expert-based journal rankings? Analysis of the Finnish publication indicator
36. Explaining deep neural networks and beyond: A review of methods and applications10.1109/JPROC.2021.3060483 ↗
37. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews10.1016/j.ijsu.2021.105906 ↗
38. Web of Science as a data source for research on scientific and scholarly activity10.1162/qss_a_00018 ↗
39. Kitchenham, B., and Charters, S. (2007). Guidelines for Performing Systematic Literature Reviews in Software Engineering, Software Engineering Group, School of Computer Science and Mathematics, Keele University. EBSE Technical Report, EBSE-2007-01.
40. Recommendation system based on deep learning methods: A systematic review and new directions10.1007/s10462-019-09744-1 ↗
41. An Interpretable Skin Cancer Classification Using Optimized Convolutional Neural Network for a Smart Healthcare System10.1109/ACCESS.2023.3269694 ↗
42. Carrieri, A.P., Haiminen, N., Maudsley-Barton, S., Gardiner, L.J., Murphy, B., Mayes, A.E., Paterson, S., Grimshaw, S., Winn, M., and Shand, C. (2021). Explainable AI reveals changes in skin microbiome composition linked to phenotypic differences. Sci. Rep., 11.10.1038/s41598-021-83922-6 ↗
43. An Explainable AI Approach for Breast Cancer Metastasis Prediction Based on Clinicopathological Data10.1109/TBME.2023.3282840 ↗
44. Yagin, B., Yagin, F.H., Colak, C., Inceoglu, F., Kadry, S., and Kim, J. (2023). Cancer Metastasis Prediction and Genomic Biomarker Identification through Machine Learning and eXplainable Artificial Intelligence in Breast Cancer Research. Diagnostics, 13.10.3390/diagnostics13213314 ↗
45. Kaplun, D., Krasichkov, A., Chetyrbok, P., Oleinikov, N., Garg, A., and Pannu, H.S. (2021). Cancer Cell Profiling Using Image Moments and Neural Networks with Model Agnostic Explainability: A Case Study of Breast Cancer Histopathological (BreakHis) Database. Mathematics, 9.10.3390/math9202616 ↗
46. Explainable artificial intelligence to predict the risk of side-specific extraprostatic extension in pre-prostatectomy patients
47. Ramirez-Mena, A., Andres-Leon, E., Alvarez-Cubero, M.J., Anguita-Ruiz, A., Martinez-Gonzalez, L.J., and Alcala-Fdez, J. (2023). Explainable artificial intelligence to predict and identify prostate cancer tissue by gene expression. Comput. Methods Programs Biomed., 240.10.1016/j.cmpb.2023.107719 ↗
48. Anjara, S.G., Janik, A., Dunford-Stenger, A., Mc Kenzie, K., Collazo-Lorduy, A., Torrente, M., Costabello, L., and Provencio, M. (2023). Examining explainable clinical decision support systems with think aloud protocols. PLoS ONE, 18.10.1371/journal.pone.0291443 ↗
49. Wani, N.A., Kumar, R., and Bedi, J. (2024). DeepXplainer: An interpretable deep learning based approach for lung cancer detection using explainable artificial intelligence. Comput. Methods Programs Biomed., 243.10.1016/j.cmpb.2023.107879 ↗
50. RoBERTa-Assisted Outcome Prediction in Ovarian Cancer Cytoreductive Surgery Using Operative Notes10.1177/10732748231209892 ↗
51. Laios, A., Kalampokis, E., Johnson, R., Munot, S., Thangavelu, A., Hutson, R., Broadhead, T., Theophilou, G., Leach, C., and Nugent, D. (2022). Factors Predicting Surgical Effort Using Explainable Artificial Intelligence in Advanced Stage Epithelial Ovarian Cancer. Cancers, 14.10.3390/cancers14143447 ↗
52. Ghnemat, R., Alodibat, S., and Abu Al-Haija, Q. (2023). Explainable Artificial Intelligence (XAI) for Deep Learning Based Medical Imaging Classification. J. Imaging, 9.10.3390/jimaging9090177 ↗
53. Unraveling COVID-19 Dynamics via Machine Learning and XAI: Investigating Variant Influence and Prognostic Classification10.3390/make5040064 ↗
54. An XAI approach for COVID-19 detection using transfer learning with X-ray images10.1016/j.heliyon.2023.e15137 ↗
55. Sargiani, V., De Souza, A.A., De Almeida, D.C., Barcelos, T.S., Munoz, R., and Da Silva, L.A. (2022). Supporting Clinical COVID-19 Diagnosis with Routine Blood Tests Using Tree-Based Entropy Structured Self-Organizing Maps. Appl. Sci., 12.10.3390/app12105137 ↗
56. CXR-Net: A Multitask Deep Learning Network for Explainable and Accurate Diagnosis of COVID-19 Pneumonia from Chest X-ray Images10.1109/JBHI.2022.3220813 ↗
57. Palatnik de Sousa, I., Vellasco, M.M.B.R., and Costa da Silva, E. (2021). Explainable Artificial Intelligence for Bias Detection in COVID CT-Scan Classifiers. Sensors, 21.10.3390/s21165657 ↗
58. Nguyen, D.Q., Vo, N.Q., Nguyen, T.T., Nguyen-An, K., Nguyen, Q.H., Tran, D.N., and Quan, T.T. (2022). BeCaked: An Explainable Artificial Intelligence Model for COVID-19 Forecasting. Sci. Rep., 12.10.1038/s41598-022-11693-9 ↗
59. Guarrasi, V., and Soda, P. (2023). Multi-objective optimization determines when, which and how to fuse deep networks: An application to predict COVID-19 outcomes. Comput. Biol. Med., 154.10.1016/j.compbiomed.2023.106625 ↗
60. A Clinical Decision Support System for Edge/Cloud ICU Readmission Model Based on Particle Swarm Optimization, Ensemble Machine Learning, and Explainable Artificial Intelligence10.1109/ACCESS.2023.3312343 ↗
61. Henzel, J., Tobiasz, J., Kozielski, M., Bach, M., Foszner, P., Gruca, A., Kania, M., Mika, J., Papiez, A., and Werner, A. (2021). Screening Support System Based on Patient Survey Data-Case Study on Classification of Initial, Locally Collected COVID-19 Data. Appl. Sci., 11.10.3390/app112210790 ↗
62. Delgado-Gallegos, J.L., Aviles-Rodriguez, G., Padilla-Rivas, G.R., Cosio-Leon, M.d.l.A., Franco-Villareal, H., Nieto-Hipolito, J.I., Lopez, J.d.D.S., Zuniga-Violante, E., Islas, J.F., and Romo-Cardenas, G.S. (2023). Application of C5.0 Algorithm for the Assessment of Perceived Stress in Healthcare Professionals Attending COVID-19. Brain Sci., 13.10.3390/brainsci13030513 ↗
63. Diagnosis of Paratuberculosis in Histopathological Images Based on Explainable Artificial Intelligence and Deep Learning10.18280/ts.390311 ↗
64. Papandrianos, I.N., Feleki, A., Moustakidis, S., Papageorgiou, I.E., Apostolopoulos, I.D., and Apostolopoulos, D.J. (2022). An Explainable Classification Method of SPECT Myocardial Perfusion Images in Nuclear Cardiology Using Deep Learning and Grad-CAM. Appl. Sci., 12.10.3390/app12157592 ↗
65. Zhang, Y., Weng, Y., and Lund, J. (2022). Applications of Explainable Artificial Intelligence in Diagnosis and Surgery. Diagnostics, 12.10.3390/diagnostics12020237 ↗
66. Rietberg, M.T., Nguyen, V.B., Geerdink, J., Vijlbrief, O., and Seifert, C. (2023). Accurate and Reliable Classification of Unstructured Reports on Their Diagnostic Goal Using BERT Models. Diagnostics, 13.10.3390/diagnostics13071251 ↗
67. Explainable Artificial Intelligence (XAI): Classification of Medical Thermal Images of Neonates Using Class Activation Maps10.18280/ts.380502 ↗
68. Dindorf, C., Konradi, J., Wolf, C., Taetz, B., Bleser, G., Huthwelker, J., Werthmann, F., Bartaguiz, E., Kniepert, J., and Drees, P. (2021). Classification and Automated Interpretation of Spinal Posture Data Using a Pathology-Independent Classifier and Explainable Artificial Intelligence (XAI). Sensors, 21.10.3390/s21186323 ↗
69. Sarp, S., Kuzlu, M., Wilson, E., Cali, U., and Guler, O. (2021). The Enlightening Role of Explainable Artificial Intelligence in Chronic Wound Classification. Electronics, 10.10.20944/preprints202101.0346.v1 ↗
70. Wang, M.H., Chong, K.K.l., Lin, Z., Yu, X., and Pan, Y. (2023). An Explainable Artificial Intelligence-Based Robustness Optimization Approach for Age-Related Macular Degeneration Detection Based on Medical IOT Systems. Electronics, 12.10.3390/electronics12122697 ↗
71. Kalyakulina, A., Yusipov, I., Kondakova, E., Bacalini, M.G., Franceschi, C., Vedunova, M., and Ivanchenko, M. (2023). Small immunological clocks identified by deep learning and gradient boosting. Front. Immunol., 14.10.3389/fimmu.2023.1177611 ↗
72. Javed, A.R., Khan, H.U., Alomari, M.K.B., Sarwar, M.U., Asim, M., Almadhor, A.S., and Khan, M.Z. (2023). Toward explainable AI-empowered cognitive health assessment. Front. Public Health, 11.10.3389/fpubh.2023.1024195 ↗
73. Valladares-Rodriguez, S., Fernandez-Iglesias, M.J., Anido-Rifon, L.E., and Pacheco-Lorenzo, M. (2022). Evaluation of the Predictive Ability and User Acceptance of Panoramix 2.0, an AI-Based E-Health Tool for the Detection of Cognitive Impairment. Electronics, 11.10.3390/electronics11213424 ↗
74. Moreno-Sanchez, P.A. (2023). Improvement of a prediction model for heart failure survival through explainable artificial intelligence. Front. Cardiovasc. Med., 10.10.3389/fcvm.2023.1219586 ↗
75. An explainable artificial intelligence-enabled electrocardiogram analysis model for the classification of reduced left ventricular function10.1093/ehjdh/ztad027 ↗
76. Explainable AI for Glaucoma Prediction Analysis to Understand Risk Factors in Treatment Planning10.1109/TIM.2022.3171613 ↗
77. Explainable framework for Glaucoma diagnosis by image processing and convolutional neural network synergy: Analysis with doctor evaluation10.1016/j.future.2021.11.018 ↗
78. Explainable Artificial Intelligence Warning Model Using an Ensemble Approach for In-Hospital Cardiac Arrest Prediction: Retrospective Cohort Study10.2196/48244 ↗
79. Obayya, M., Nemri, N., Nour, M.K., Al Duhayyim, M., Mohsen, H., Rizwanullah, M., Zamani, A.S., and Motwakel, A. (2022). Explainable Artificial Intelligence Enabled TeleOphthalmology for Diabetic Retinopathy Grading and Classification. Appl. Sci., 12.10.3390/app12178749 ↗
80. Explainable Artificial Intelligence (XAI) for the Prediction of Diabetes Management: An Ensemble Approach
81. A Mobile App That Addresses Interpretability Challenges in Machine Learning-Based Diabetes Predictions: Survey-Based User Study10.2196/50328 ↗
82. Maaroof, N., Moreno, A., Valls, A., Jabreel, M., and Romero-Aroca, P. (2023). Multi-Class Fuzzy-LORE: A Method for Extracting Local and Counterfactual Explanations Using Fuzzy Decision Trees. Electronics, 12.10.3390/electronics12102215 ↗
83. Designing ECG monitoring healthcare system with federated transfer learning and explainable AI10.1016/j.knosys.2021.107763 ↗
84. Interpretation and Classification of Arrhythmia Using Deep Convolutional Network10.1109/TIM.2022.3204316 ↗
85. Mollaei, N., Fujao, C., Silva, L., Rodrigues, J., Cepeda, C., and Gamboa, H. (2022). Human-Centered Explainable Artificial Intelligence: Automotive Occupational Health Protection Profiles in Prevention Musculoskeletal Symptoms. Int. J. Environ. Res. Public Health, 19.10.3390/ijerph19159552 ↗
86. Petrauskas, V., Jasinevicius, R., Damuleviciene, G., Liutkevicius, A., Janaviciute, A., Lesauskaite, V., Knasiene, J., Meskauskas, Z., Dovydaitis, J., and Kazanavicius, V. (2021). Explainable Artificial Intelligence-Based Decision Support System for Assessing the Nutrition-Related Geriatric Syndromes. Appl. Sci., 11.10.3390/app112411763 ↗
87. Ensuring fair, safe, and interpretable artificial intelligence-based prediction tools in a real-world oncological setting10.1038/s43856-023-00317-6 ↗
88. Quality medical data management within an open AI architecture-cancer patients case10.1080/09540091.2023.2194581 ↗
89. Zhang, H., and Ogasawara, K. (2023). Grad-CAM-Based Explainable Artificial Intelligence Related to Medical Text Processing. Bioengineering, 10.10.3390/bioengineering10091070 ↗
90. Zlahtic, B., Zavrsnik, J., Vosner, H.B., Kokol, P., Suran, D., and Zavrsnik, T. (2023). Agile Machine Learning Model Development Using Data Canyons in Medicine: A Step towards Explainable Artificial Intelligence and Flexible Expert-Based Model Improvement. Appl. Sci., 13.10.3390/app13148329 ↗
91. Gouverneur, P., Li, F., Shirahama, K., Luebke, L., Adamczyk, W.M., Szikszay, T.M.M., Luedtke, K., and Grzegorzek, M. (2023). Explainable Artificial Intelligence (XAI) in Pain Research: Understanding the Role of Electrodermal Activity for Automated Pain Recognition. Sensors, 23.10.3390/s23041959 ↗
92. Real, K.S.D., and Rubio, A. (2023). Discovering the mechanism of action of drugs with a sparse explainable network. Ebiomedicine, 95.10.1016/j.ebiom.2023.104767 ↗
93. Park, A., Lee, Y., and Nam, S. (2023). A performance evaluation of drug response prediction models for individual drugs. Sci. Rep., 13.10.1038/s41598-023-39179-2 ↗
94. A Trustworthy View on Explainable Artificial Intelligence Method Evaluation10.1109/MC.2022.3233806 ↗
95. Evaluating the sustainability of smart technology applications in healthcare after the COVID-19 pandemic: A hybridising subjective and objective fuzzy group decision-making approach with explainable artificial intelligence
96. Bhatia, S., and Albarrak, A.S. (2023). A Blockchain-Driven Food Supply Chain Management Using QR Code and XAI-Faster RCNN Architecture. Sustainability, 15.10.3390/su15032579 ↗
97. Konradi, J., Zajber, M., Betz, U., Drees, P., Gerken, A., and Meine, H. (2022). AI-Based Detection of Aspiration for Video-Endoscopy with Visual Aids in Meaningful Frames to Interpret the Model Outcome. Sensors, 22.10.20944/preprints202212.0051.v1 ↗
98. Aquino, G., Costa, M.G.F., and Costa Filho, C.F.F. (2023). Explaining and Visualizing Embeddings of One-Dimensional Convolutional Models in Human Activity Recognition Tasks. Sensors, 23.10.3390/s23094409 ↗
99. Hardware Implementation for Lower Limb Surface EMG Measurement and Analysis Using Explainable AI for Activity Recognition10.1109/TIM.2022.3198443 ↗
100. Iliadou, E., Su, Q., Kikidis, D., Bibas, T., and Kloukinas, C. (2022). Profiling hearing aid users through big data explainable artificial intelligence techniques. Front. Neurol., 13.10.3389/fneur.2022.933940 ↗