journal article·2022-01-27Open access

A Systematic Review of Explainable Artificial Intelligence in Terms of Different Application Domains and Tasks

Mir Riyanul Islam, Mobyen Uddin Ahmed, Shaibal Barua, Shahina Begum

Applied Sciences· Volume 12· Issue 3

Abstract

Artificial intelligence (AI) and machine learning (ML) have recently been radically improved and are now being employed in almost every application domain to develop automated or semi-automated systems. To facilitate greater human acceptability of these systems, explainable artificial intelligence (XAI) has experienced significant growth over the last couple of years with the development of highly accurate models but with a paucity of explainability and interpretability. The literature shows evidence from numerous studies on the philosophy and methodologies of XAI. Nonetheless, there is an evident scarcity of secondary studies in connection with the application domains and tasks, let alone review studies following prescribed guidelines, that can enable researchers’ understanding of the current trends in XAI, which could lead to future research for domain- and application-specific method development. Therefore, this paper presents a systematic literature review (SLR) on the recent developments of XAI methods and evaluation metrics concerning different application domains and tasks. This study considers 137 articles published in recent years and identified through the prominent bibliographic databases. This systematic synthesis of research articles resulted in several analytical findings: XAI methods are mostly developed for safety-critical domains worldwide, deep learning and ensemble models are being exploited more than other types of AI/ML models, visual explanations are more acceptable to end-users and robust evaluation metrics are being developed to assess the quality of explanations. Research studies have been performed on the addition of explanations to widely used AI/ML models for expert users. However, more attention is required to generate explanations for general users from sensitive domains such as finance and the judicial system.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Explainable Artificial Intelligence (XAI) · Adversarial Robustness in Machine Learning · Artificial Intelligence in Healthcare and Education

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Artificial Intelligence and Intelligent Systems Research Group, School of Innovation, Design and Engineering, Mälardalen University, Högskoleplan 1, 722 20 Västerås, Sweden

Funding

European Union · Vetenskapsrådet - The Swedish Research Council

References (100+)

Show source-provided references
1. Explainable AI: From Black Box to Glass Box10.1007/s11747-019-00710-5 ↗
2. A Survey of Methods for Explaining Black Box Models10.1145/3236009 ↗
3. Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View10.1109/ACCESS.2019.2949286 ↗
4. Enhanced Maintenance and Explanation of Expert Systems Through Explicit Models of Their Development10.1109/TSE.1985.231882 ↗
5. DARPA’s Explainable Artificial Intelligence (XAI) Program
6. Explainable AI: A Brief Survey on History, Research Areas, Approaches and Challenges10.1007/978-3-030-32236-6_51 ↗
7. Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR
8. Vilone, G., and Longo, L. (2020). Explainable Artificial Intelligence: A Systematic Review. arXiv.
9. Classification of Explainable Artificial Intelligence Methods through Their Output Formats10.3390/make3030032 ↗
10. A Review of Explanation Methods for Bayesian Networks10.1017/S026988890200019X ↗
11. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 23). Model-Agnostic Interpretability of Machine Learning. Proceedings of the ICML Workshop on Human Interpretability in Machine Learning, New York, NY, USA. Available online: https://arxiv.org/abs/1606.05386.
12. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13–17). “Why Should I Trust You?” Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939778 ↗
13. A Bibliometric Analysis of the Explainable Artificial Intelligence Research Field
14. Explainable AI: The New 42?10.1007/978-3-319-99740-7_21 ↗
15. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead10.1038/s42256-019-0048-x ↗
16. Dosilovic, F.K., Brcic, M., and Hlupic, N. (2018, January 21–25). Explainable Artificial Intelligence: A Survey. Proceedings of the 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO 2018), Opatija, Croatia.10.23919/MIPRO.2018.8400040 ↗
17. Mittelstadt, B., Russell, C., and Wachter, S. (2019, January 29–31). Explaining Explanations in AI. Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* 2019), Atlanta, GA, USA.10.1145/3287560.3287574 ↗
18. Towards Explainable Artificial Intelligence
19. Preece, A., Harborne, D., Braines, D., Tomsett, R., and Chakraborty, S. (2018, January 18–20). Stakeholders in Explainable AI. Proceedings of the AAAI FSS-18: Artificial Intelligence in Government and Public Sector, Arlington, VA, USA. Available online: https://arxiv.org/abs/1810.00184.
20. Explainable Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI10.1016/j.inffus.2019.12.012 ↗
21. Explainable Artificial Intelligence: Concepts, Applications, Research Challenges and Visions10.1007/978-3-030-57321-8_1 ↗
22. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)10.1109/ACCESS.2018.2870052 ↗
23. Factual and Counterfactual Explanations for Black Box Decision Making10.1109/MIS.2019.2957223 ↗
24. Zhou, J., and Chen, F. (2018). Perturbation-Based Explanations of Prediction Models. Human and Machine Learning, Springer International Publishing.10.1007/978-3-319-90403-0 ↗
25. Zhang, Q., Wu, Y.N., and Zhu, S.C. (2018, January June). Interpretable Convolutional Neural Networks. Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00920 ↗
26. Aceves-Fernandez, M.A. (2020). Explainable Artificial Intelligence (xAI) Approaches and Deep Meta-Learning Models. Advances and Applications in Deep Learning, InTechOpen.10.5772/intechopen.87786 ↗
27. Notions of explainability and evaluation approaches for explainable artificial intelligence10.1016/j.inffus.2021.05.009 ↗
28. Causability and Explainability of Artificial Intelligence in Medicine
29. Explainable Artificial Intelligence Applications in NLP, Biomedical, and Malware Classification: A Literature Review10.1007/978-3-030-22868-2_90 ↗
30. Explainable Artificial Intelligence for Neuroscience: Behavioral Neurostimulation10.3389/fnins.2019.01346 ↗
31. Drug Discovery with Explainable Artificial Intelligence10.1038/s42256-020-00236-4 ↗
32. Explainable Artificial Intelligence Models using Real-world Electronic Health Record Data: A Systematic Scoping Review10.1093/jamia/ocaa053 ↗
33. Artificial Intelligence, Machine Learning and Reasoning in Health Informatics—Case Studies
34. Gulum, M.A., Trombley, C.M., and Kantardzic, M. (2021). A Review of Explainable Deep Learning Cancer Detection Models in Medical Imaging. Appl. Sci., 11.10.3390/app11104573 ↗
35. Gade, K., Geyik, S.C., Kenthapadi, K., Mithal, V., and Taly, A. (2019, January 4–8). Explainable AI in Industry. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Anchorage, AK, USA.10.1145/3292500.3332281 ↗
36. Dam, H.K., Tran, T., and Ghose, A. (June, January 27). Explainable Software Analytics. Proceedings of the 40th International Conference on Software Engineering New Ideas and Emerging Results–ICSE-NIER ’18, Gothenburg, Sweden.
37. Exploration of Explainable AI in Context of Human–Machine Interface for the Assistive Driving System
38. Kitchenham, B., and Charters, S. (2007). Guidelines for Performing Systematic Literature Reviews in Software Engineering, Durham University. Technical Report.
39. Guidelines for Performing Systematic Research Projects Reviews
40. Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., and The PRISMA Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med., 6.10.1371/journal.pmed.1000097 ↗
41. Recommendation System based on Deep Learning Methods: A Systematic Review and New Directions10.1007/s10462-019-09744-1 ↗
42. A Systematic Literature Review: Opinion Mining Studies from Mobile App Store User Reviews10.1016/j.jss.2016.11.027 ↗
43. Wohlin, C. (2014, January 13–14). Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering. Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering–EASE ’14, London, UK.10.1145/2601248.2601268 ↗
44. Exploration of Unstructured Nrrative Crime Reports: An Unsupervised Neural Network and Point Pattern Analysis Approach10.1080/15230406.2013.779780 ↗
45. Tintarev, N., Rostami, S., and Smyth, B. (2018, January 9–13). Knowing the unknown: Visualising consumption blind-spots in recommender systems. Proceedings of the 33rd Annual ACM Symposium on Applied Computing (SAC’18), Pau, France.10.1145/3167132.3167419 ↗
46. Galhotra, S., Pradhan, R., and Salimi, B. (2021, January 20–25). Explaining Black-Box Algorithms Using Probabilistic Contrastive Counterfactuals. Proceedings of the 2021 International Conference on Management of Data, Virtual Event.10.1145/3448016.3458455 ↗
47. CASTLE: Cluster-Aided Space Transformation for Local Explanations10.1016/j.eswa.2021.115045 ↗
48. PASTLE: Pivot-Aided Space Transformation for Local Explanations10.1016/j.patrec.2021.05.018 ↗
49. Post-hoc Explanation of Black-box Classifiers using Confident Itemsets10.1016/j.eswa.2020.113941 ↗
50. Hatwell, J., Gaber, M.M., and Muhammad Atif Azad, R. (2021). Gbt-hips: Explaining the classifications of gradient boosted tree ensembles. Appl. Sci., 11.10.3390/app11062511 ↗
51. Rubio-Manzano, C., Segura-Navarrete, A., Martinez-Araneda, C., and Vidal-Castro, C. (2021). Explainable hopfield neural networks using an automatic video-generation system. Appl. Sci., 11.10.3390/app11135771 ↗
52. Alonso, J.M., Toja-Alamancos, J., and Bugarin, A. (2020, January 19–24). Experimental Study on Generating Multi-modal Explanations of Black-box Classifiers in terms of Gray-box Classifiers. Proceedings of the 2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Glasgow, UK.10.1109/FUZZ48607.2020.9177770 ↗
53. Towards Explanatory Interactive Image Captioning Using Top-Down and Bottom-Up Features, Beam Search and Re-ranking10.1007/s13218-020-00679-2 ↗
54. Learning Explainable Decision Rules via Maximum Satisfiability10.1109/ACCESS.2020.3041040 ↗
55. Random Forest Explainability using Counterfactual Sets10.1016/j.inffus.2020.07.001 ↗
56. Measuring the Quality of Explanations: The System Causability Scale (SCS)10.1007/s13218-020-00636-z ↗
57. A Robust Algorithm for Explaining Unreliable Machine Learning Survival Models using Kolmogorov–Smirnov Bounds10.1016/j.neunet.2020.08.007 ↗
58. Le, T., Wang, S., and Lee, D. (2020, January 23–27). GRACE: Generating Concise and Informative Contrastive Sample to Explain Neural Network Model’s Prediction. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, Virtual Event.10.1145/3394486.3403066 ↗
59. From Local Explanations to Global Understanding with Explanable AI for Trees10.1038/s42256-019-0138-9 ↗
60. Enhancing Explainability of Neural Networks Through Architecture Constraints
61. Semantically Explainable Fuzzy Classifier10.1142/S0218001420510064 ↗
62. Holzinger, A., Kieseberg, P., Tjoa, A.M., and Weippl, E. (2018). Evaluating Explanations by Cognitive Value. Machine Learning and Knowledge Extraction, Springer International Publishing.10.1007/978-3-319-99740-7 ↗
63. Comparison-Based Inverse Classification for Interpretability in Machine Learning
64. Pierrard, R., Poli, J.P., and Hudelot, C. (2018, January 8–13). Learning Fuzzy Relations and Properties for Explainable Artificial Intelligence. Proceedings of the 2018 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Rio de Janeiro, Brazil.10.1109/FUZZ-IEEE.2018.8491538 ↗
65. Plumb, G., Molitor, D., and Talwalkar, A. (2018, January 3–8). Model Agnostic Supervised Local Explanations. Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS ’18), Montreal, QC, Canada.
66. Hanratty, T.P., and Llinas, J. (2017). An Apporach to Explainable Deep Learning using Fuzzy Inference. Next-Generation Analyst V, SPIE.
67. Explaining Prediction Models and Individual Predictions with Feature Contributions10.1007/s10115-013-0679-x ↗
68. A Methodology to Explain Neural Network Classification10.1016/S0893-6080(01)00127-7 ↗
69. Chandrasekaran, J., Lei, Y., Kacker, R., and Richard Kuhn, D. (2021, January 12–16). A Combinatorial Approach to Explaining Image Classifiers. Proceedings of the 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Virtual Event.10.1109/ICSTW52544.2021.00019 ↗
70. Explaining CNN and RNN Using Selective Layer-Wise Relevance Propagation10.1109/ACCESS.2021.3051171 ↗
71. Mitigating Belief Projection in Explainable Artificial Intelligence via Bayesian Teaching10.1038/s41598-021-89267-4 ↗
72. Schorr, C., Goodarzi, P., Chen, F., and Dahmen, T. (2021). Neuroscope: An Explainable AI Toolbox for Semantic Segmentation and Image Classification of Convolutional Neural Nets. Appl. Sci., 11.10.3390/app11052199 ↗
73. Towards Explainable Deep Neural Networks (xDNN)10.1016/j.neunet.2020.07.010 ↗
74. Middle-Level Features for the Explanation of Classification Systems by Sparse Dictionary Methods10.1142/S0129065720500409 ↗
75. An Adversarial Explainable Artificial Intelligence (XAI) based Approach for Action Forecasting
76. Information Fusion-2-Text: Explainable Aggregation via Linguistic Protoforms
77. Salient Explanation for Fine-Grained Classification10.1109/ACCESS.2020.2980742 ↗
78. Poyiadzi, R., Sokol, K., Santos-Rodriguez, R., De Bie, T., and Flach, P. (2020, January 7–8). FACE: Feasible and Actionable Counterfactual Explanations. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, New York, NY, USA.10.1145/3375627.3375850 ↗
79. Explaining VQA Predictions using Visual Grounding and a Knowledge Base10.1016/j.imavis.2020.103968 ↗
80. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization10.1007/s11263-019-01228-7 ↗
81. Tan, R., Khan, N., and Guan, L. (2020, January 19–24). Locality Guided Neural Networks for Explainable Artificial Intelligence. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.10.1109/IJCNN48605.2020.9207559 ↗
82. Interpretable Deep Convolutional Fuzzy Classifier
83. Oramas M., J., Wang, K., and Tuytelaars, T. (2019, January 6–9). Visual Explanation by Interpretation: Improving Visual Feedback Capabilities of Deep Neural Networks. Proceedings of the International Conference on Learning Representations (ICLR 2019), New Orleans, LA, USA.
84. Lundberg, S.M., and Lee, S.I. (2017, January 4–9). A Unified Approach to Interpreting Model Predictions. Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS’17), Long Beach, CA, USA.
85. Explaining Nonlinear Classification Decisions with Deep Taylor Decomposition10.1016/j.patcog.2016.11.008 ↗
86. Generating Visual Explanations10.1007/978-3-319-46493-0_1 ↗
87. Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., and Samek, W. (2015). On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLoS ONE, 10.10.1371/journal.pone.0130140 ↗
88. Alonso, J.M., Ducange, P., Pecori, R., and Vilas, R. (2020, January 19–24). Building Explanations for Fuzzy Decision Trees with the ExpliClas Software. Proceedings of the 2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Glasgow, UK.10.1109/FUZZ48607.2020.9177725 ↗
89. Explainable AI: A Hybrid Approach to Generate Human-Interpretable Explanation for Deep Learning Prediction10.1016/j.procs.2020.02.255 ↗
90. Enabling Explainable Fusion in Deep Learning with Fuzzy Integral Neural Networks10.1109/TFUZZ.2019.2917124 ↗
91. Meskauskas, Z., Jasinevicius, R., Kazanavicius, E., and Petrauskas, V. (2020, January 19–24). XAI-Based Fuzzy SWOT Maps for Analysis of Complex Systems. Proceedings of the 2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Glasgow, UK.10.1109/FUZZ48607.2020.9177792 ↗
92. Interpretable Confidence Measures for Decision Support Systems10.1016/j.ijhcs.2020.102493 ↗
93. Human-Centric AI for Trustworthy IoT Systems With Explainable Multilayer Perceptrons10.1109/ACCESS.2019.2937521 ↗
94. RuleMatrix: Visualizing and Understanding Classifiers with Rules10.1109/TVCG.2018.2864812 ↗
95. Magdalena, L. (2018, January 8–13). Designing Interpretable Hierarchical Fuzzy Systems. Proceedings of the 2018 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), Rio de Janeiro, Brazil.10.1109/FUZZ-IEEE.2018.8491452 ↗
96. Ribeiro, M.T., Singh, S., and Guestrin, C. (2018, January 2–7). Anchors: High-precision Model-agnostic Explanations. Proceedings of the 32nd AAAI Conference on Artificial Intelligence, (AAAI-18), New Orleans, LA, USA.10.1609/aaai.v32i1.11491 ↗
97. Massie, S., Craw, S., and Wiratunga, N. (2004). A Visualisation Tool to Explain Case-Base Reasoning Solutions for Tablet Formulation. Applications and Innovations in Intelligent Systems XII, Springer.
98. Interpretable Neural Networks based on Continuous-valued Logic and Multicriteria Decision Operators10.1016/j.knosys.2020.105972 ↗
99. An Information-Theoretic Approach to Personalized Explainable Machine Learning10.1109/LSP.2020.2993176 ↗
100. Generating and Understanding Personalized Explanations in Hybrid Recommender Systems10.1145/3365843 ↗