Making Sense of Machine Learning: A Review of Interpretation Techniques and Their Applications
Source abstract
Abstract
Transparency in AI models is essential for promoting human–AI collaboration and ensuring regulatory compliance. However, interpreting these models is a complex process influenced by various methods and datasets. This study presents a comprehensive overview of foundational interpretation techniques, meticulously referencing the original authors and emphasizing their pivotal contributions. Recognizing the seminal work of these pioneers is imperative for contextualizing the evolutionary trajectory of interpretation in the field of AI. Furthermore, this research offers a retrospective analysis of interpretation techniques, critically evaluating their inherent strengths and limitations. We categorize these techniques into model-based, representation-based, post hoc, and hybrid methods, delving into their diverse applications. Furthermore, we analyze publication trends over time to see how the adoption of advanced computational methods within various categories of interpretation techniques has shaped the development of AI interpretability over time. This analysis highlights a notable preference shift towards data-driven approaches in the field. Moreover, we consider crucial factors such as the suitability of these techniques for generating local or global insights and their compatibility with different data types, including images, text, and tabular data. This structured categorization serves as a guide for practitioners navigating the landscape of interpretation techniques in AI. In summary, this review not only synthesizes various interpretation techniques but also acknowledges the contributions of their original authors. By emphasizing the origins of these techniques, we aim to enhance AI model explainability and underscore the importance of recognizing biases, uncertainties, and limitations inherent in the methods and datasets. This approach promotes the ethical and practical use of interpretation insights, empowering AI practitioners, researchers, and professionals to make informed decisions when selecting techniques for responsible AI implementation in real-world scenarios.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Explainable Artificial Intelligence (XAI) · Adversarial Robustness in Machine Learning · Machine Learning in Healthcare
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Data61, Commonwealth Scientific and Industrial Research Organisation (CSIRO), Canberra, ACT 2601, Australia
References (96)
Show source-provided references
1. Black is the new orange: How to determine AI liability10.1007/s10506-022-09308-9 ↗
2. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI10.1016/j.inffus.2019.12.012 ↗
3. Gunning, D., Vorm, E., Wang, Y.J., and Turek, M. (2021). DARPA’s explainable AI (XAI) program: A retrospective. Appl. AI Lett., 2.10.1002/ail2.61 ↗
4. Abdullah, T.A.A., Zahid, M.S.M., and Ali, W. (2021). A Review of Interpretable ML in Healthcare: Taxonomy, Applications, Challenges, and Future Directions. Symmetry, 13.10.3390/sym13122439 ↗
5. Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications10.1109/JPROC.2021.3060483 ↗
6. Definitions, methods, and applications in interpretable machine learning10.1073/pnas.1900654116 ↗
7. Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., and Kagal, L. (2018, January 1–3). Explaining Explanations: An Overview of Interpretability of Machine Learning. Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy. Available online: https://api.semanticscholar.org/CorpusID:59600034.10.1109/DSAA.2018.00018 ↗
8. (2023, February 16). The Lisp Approach to AI. Available online: https://medium.com/ai-society/the-lisp-approach-to-ai-part-1-a48c7385a913.
9. (2023, February 16). How Lisp Became God’s Own Programming Language. Available online: https://twobithistory.org/2018/10/14/lisp.html.
10. (2023, February 18). ELIZA on Wikipedia. Available online: https://en.wikipedia.org/wiki/ELIZA.
11. (2023, May 01). OpenAI. ChatGPT (Mar 14 Version) [Large Language Model]. Available online: https://chat.openai.com/chat.
12. Wikipedia (2023, January 01). MAD (Programming Language). Available online: https://en.wikipedia.org/wiki/MAD_(programming_language).
13. Wikipedia (2023, November 01). Universal Approximation Theorem. Available online: https://en.wikipedia.org/wiki/Universal_approximation_theorem.
14. (2023, February 16). Paul Workman, Reflecting on DeepMind’s AlphaFold Artificial Intelligence Success—What’s the Real Significance for Protein Folding Research and Drug Discovery. The Institute of Cancer Research. Available online: https://www.icr.ac.uk/blogs/the-drug-discoverer/page-details/reflecting-on-deepmind-s-alphafold-artificial-intelligence-success-what-s-the-real-significance-for-protein-folding-research-and-drug-discovery.
15. Lundberg, S.M., and Lee, S.-I. (2017, January 4–9). A Unified Approach to Interpreting Model Predictions. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, Long Beach, CA, USA. Available online: https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd28b67767-Abstract.html.
16. Krishnapuram, B., Shah, M., Smola, A.J., Aggarwal, C.C., Shen, D., and Rastogi, R. (2016, January 13–17). “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.
17. Regression Towards Mediocrity in Hereditary Stature
18. McCarthy, J. (1960). Recursive Functions of Symbolic Expressions: Their Computation by Machine, Part I, Massachusetts Institute of Technology.10.1145/367177.367199 ↗
19. Induction of Decision Trees10.1007/BF00116251 ↗
20. Ribeiro, M.T., Singh, S., and Guestrin, C. (2018, January 2–7). Anchors: High-Precision Model-Agnostic Explanations. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11491 ↗
21. Lundberg, S.M., Erion, G.G., and Lee, S.I. (2019). Consistent Individualized Feature Attribution for Tree Ensembles. arXiv.
22. Shrikumar, A., Greenside, P., and Kundaje, A. (2019). Learning Important Features Through Propagating Activation Differences. arXiv.
23. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization10.1007/s11263-019-01228-7 ↗
24. Evolutionary Fuzzy Systems for Explainable Artificial Intelligence: Why, When, What for, and Where to?10.1109/MCI.2018.2881645 ↗
25. A Bayesian framework for learning rule sets for interpretable classification
26. Wang, T., Rudin, C., Velez-Doshi, F., Liu, Y., Klampfl, E., and MacNeille, P. (2016, January 12–15). Bayesian Rule Sets for Interpretable Classification. Proceedings of the 2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, Spain.10.1109/ICDM.2016.0171 ↗
27. Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model10.1214/15-AOAS848 ↗
28. From Local Explanations to Global Understanding with Explainable AI for Trees10.1038/s42256-019-0138-9 ↗
29. Calvaresi, D., Najjar, A., Schumacher, M., and Främling, K. (2019). Explainable, Transparent Autonomous Agents and Multi-Agent Systems, Springer International Publishing.10.1007/978-3-030-30391-4 ↗
30. Norouzi, M., Collins, M.D., Johnson, M., Fleet, D.J., and Kohli, P. (2015, January 7–12). Efficient Non-greedy Optimization of Decision Trees. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Montreal, QC, Canada.
31. Mishra, P. (2022). Practical Explainable AI Using Python, APress Media, LLC, Part of Springer Nature.10.1007/978-1-4842-7158-2 ↗
32. Recent advances in decision trees: An updated survey10.1007/s10462-022-10275-5 ↗
33. Random Forests10.1023/A:1010933404324 ↗
34. Application of the logistic function to bioassay
35. Linear regression analysis—Part 14 of a series on evaluation of scientific publications
36. An Introduction to Logistic Regression Analysis and Reporting10.1080/00220670209598786 ↗
37. Berk, R.A. (2023, January 01). An Introduction to Ensemble Methods for Data Analysis (arXiv:2110.01889). UCLA, Department of Statistics Papers. Available online: https://escholarship.org/content/qt54d6g9gf/qt54d6g9gf.pdf.
38. Seni, G., and Elder, J.F. (2010). Ensemble Methods in Data Mining: Improving Accuracy through Combining Predictions, Morgan & Claypool Publishers LLC.10.1007/978-3-031-01899-2 ↗
39. Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., and Tarantola, S. (2008). Global Sensitivity Analysis: The Primer, John Wiley & Sons.10.1002/9780470725184 ↗
40. Global sensitivity analysis-enhanced surrogate (GSAS) modeling for reliability analysis10.1007/s00158-015-1347-4 ↗
41. Greedy Function Approximation: A Gradient Boosting Machine10.1214/aos/1013203451 ↗
42. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems, Morgan Kaufmann. Available online: https://www.sciencedirect.com/science/article/pii/B9780080514895500096.
43. Pearl, J. (2011). Bayesian Networks, Department of Statistics, UCLA. Available online: https://escholarship.org/uc/item/53n4f34m.
44. Russell, S.J., and Norvig, P. (2003). Artificial Intelligence: A Modern Approach, Pearson Education. Available online: http://portal.acm.org/citation.cfm?id=773294.
45. A Model of Saliency-Based Visual Attention for Rapid Scene Analysis10.1109/34.730558 ↗
46. Erhan, D., Bengio, Y., Courville, A., and Vincent, P. (2009). Visualizing Higher-Layer Features of a Deep Network, University of Montreal.
47. Morch, N.J.S., Kjems, U., Hansen, L.K., Svarer, C., Law, I., Lautrup, B., Strother, S., and Rehm, K. (December, January 27). Visualization of neural networks using saliency maps. Proceedings of the ICNN’95-International Conference on Neural Networks, Perth, WA, Australia.
48. Simonyan, K., Vedaldi, A., and Zisserman, A. (2013). Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv.
49. Zahavy, T., Ben-Zrihem, N., and Mannor, S. (2016, January 16–24). Graying the black box: Understanding DQNs. Proceedings of the 33rd International Conference on Machine Learning (PMLR), New York City, NY, USA.
50. Kindermans, P.-J., Hooker, S., Adebayo, J., Brain, G., Alber, M., Schütt, K., Tu-Berlin, S., Erhan, D., and Brain, K. (2017). Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer.
51. Rigotti, M., Miksovic, C., Giurgiu, I., Gschwind, T., and Scotton, P. (2023, January 01). Attention-based Interpretability with Concept Transformers. International Conference on Learning Representations. Available online: https://www.matrig.net/publications/articles/rigotti2022.pdf.
52. Ghaeini, R., Fern, X., and Tadepalli, P. (November, January 31). Interpreting Recurrent and Attention-Based Neural Models: A Case Study on Natural Language Inference. Proceedings of the Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.
53. Jain, S., and Wallace, B. (2019). Attention is not Explanation. arXiv.
54. Neely, M., Schouten, S., Bleeker, M., and Lucic, A. (2021). Order in the Court: Explainable AI Methods Prone to Disagreement. arXiv.
55. Serrano, S., and Smith, N. (2019). Is Attention Interpretable?. arXiv.10.18653/v1/P19-1282 ↗
56. Mylonas, N., Mollas, I., and Tsoumakas, G. (2022). Improving attention-based interpretability of text classification transformers: A preprint. arXiv.
57. Analysis of regression in game theory approach10.1002/asmb.446 ↗
58. Comparison of Contextual Importance and Utility with LIME and Shapley Values10.1007/978-3-030-82017-6_3 ↗
59. Adversarial explanations for understanding image classification decisions and improved neural network robustness10.1038/s42256-019-0104-6 ↗
60. Permutation importance: A corrected feature importance measure10.1093/bioinformatics/btq134 ↗
61. Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (2015). Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada, 7–12 December 2015, Curran Associates, Inc.. Available online: https://proceedings.neurips.cc/paper/2015/file/82965d4ed8150294d4330ace00821d77-Paper.pdf.
62. Singh, V., Cyras, K., and Inam, R. (2022). Practical Explainable AI Using Python, APress Media, LLC, Part of Springer Nature.
63. Jentzsch, S.F., Höhn, S., and Hochgeschwender, N. (2019). Practical Explainable AI Using Python, APress Media, LLC, Part of Springer Nature.
64. Keane, M.T., and Kenny, E.M. (2019). Case-Based Reasoning Research and Development, Springer International Publishing.
65. La Malfa, E., Zbrzezny, A., Michelmore, R., Paoletti, N., and Kwiatkowska, M. (2021). On guaranteed optimal robust explanations for NLP models. arXiv.10.24963/ijcai.2021/366 ↗
66. Ignatiev, A. (2023, January 01). Towards Trustable Explainable AI. Electronic proceedings of IJCAI 2020. Available online: https://www.ijcai.org/proceedings/2020/726.10.24963/ijcai.2020/726 ↗
67. Amazon (2023, November 07). Machine Learning Service—Amazon SageMaker. Available online: https://aws.amazon.com/pm/sagemaker/.
68. Google (2023, November 07). Introduction to Vertex Explainable AI. Available online: https://cloud.google.com/vertex-ai/docs/explainable-ai/overview.
69. Microsoft (2023, November 07). Azure Machine Learning. Available online: https://azure.microsoft.com/en-au/products/machine-learning.
70. IBM (2023, November 07). IBM Watson Studio. Available online: https://www.ibm.com/products/watson-studio.
71. The Linux Foundation (2023, November 07). AI Explainability 360: Understand How ML Models Predict Labels. Available online: https://ai-explainability-360.org/.
72. Predicting outcomes of hormone and chemotherapy in the molecular taxonomy of breast cancer international consortium (METABRIC) study by biochemically-inspired machine learning10.12688/f1000research.9417.3 ↗
73. Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., and Summers, R.M. (2019, January 15–20). ChestX-ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.
74. Scalable and accurate deep learning with electronic health records10.1038/s41746-018-0029-1 ↗
75. Explainable Machine Learning in Credit Risk Management10.1007/s10614-020-10042-0 ↗
76. Kumar, S., Vishal, M., and Ravi, V. (2022). Explainable Reinforcement Learning on Financial Stock Trading using SHAP. arXiv.
77. Ji, Y. (2021). Explainable AI Methods for Credit Card Fraud Detection: Evaluation of LIME and SHAP through a User Study. [Master’s Thesis, University of Skövde, School of Informatics].
78. Hybrid interpretable predictive machine learning model for air pollution prediction10.1016/j.neucom.2021.09.051 ↗
79. Using Explainable Machine Learning Forecasts to Discover Subseasonal Drivers of High Summer Temperatures in Western and Central Europe10.1175/MWR-D-21-0201.1 ↗
80. Wu, X., Zhang, Q., Wen, F., and Qi, Y. (2022). A Water Quality Prediction Model Based on Multi-Task Deep Learning: A Case Study of the Yellow River, China. Water, 14.10.3390/w14213408 ↗
81. Interpretation of ensemble learning to predict water quality using explainable artificial intelligence10.1016/j.scitotenv.2022.155070 ↗
82. Dressel, J., and Farid, H. (2018). The accuracy, fairness, and limits of predicting recidivism. Sci. Adv., 4.10.1126/sciadv.aao5580 ↗
83. Kusner, M.J., Loftus, J., Russell, C., and Silva, R. (2017, January 4–9). Counterfactual Fairness. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
84. Rudin, C., and Radin, J. (2019). Why are we using black box models in AI when we don’t need to? A lesson from an explainable AI competition. Harv. Data Sci. Rev., 1.10.1162/99608f92.5a8a3a3d ↗
85. Lipton, Z.C. (2018). The Mythos of Model Interpretability. arXiv.10.1145/3233231 ↗
86. Toward explainable and advisable model for self-driving cars
87. Keneni, B.M. (2018). Evolving Rule-Based Explainable Artificial Intelligence for Decision Support System of Unmanned Aerial Vehicles. [Master’s Thesis, University of Toledo]. Available online: http://rave.ohiolink.edu/etdc/view?acc_num=toledo1525094091882295.
88. Calvaresi, D., Najjar, A., Winikoff, M., and Främling, K. (2020). Explainable, Transparent Autonomous Agents and Multi-Agent Systems, Springer International Publishing. Available online: http://link.springer.com/10.1007/978-3-030-51924-7_4.10.1007/978-3-030-51924-7 ↗
89. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead10.1038/s42256-019-0048-x ↗
90. The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable AI10.1016/j.ijhcs.2020.102551 ↗
91. DARPA’s Explainable Artificial Intelligence Program
92. Liao, Q.V., Gruen, D., and Miller, S. (2020, January 25–30). Questioning the AI: Informing Design Practices for Explainable AI User Experiences. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA. Available online: https://api.semanticscholar.org/CorpusID:210064344.10.1145/3313831.3376590 ↗
93. Borisov, V., Leemann, T., Seßler, K., Haug, J., Pawelczyk, M., and Kasneci, G. (2022). Deep Neural Networks and Tabular Data: A Survey. arXiv, Available online: http://arxiv.org/abs/2110.01889.10.1109/TNNLS.2022.3229161 ↗
94. Fisher, A., Rudin, C., and Dominici, F. (2019). All Models Are Wrong, but Many Are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously. arXiv.
95. Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., and Zhong, C. (2021). Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges. arXiv.10.1214/21-SS133 ↗
96. Guo, C., Pleiss, G., Sun, Y., and Weinberger, K.Q. (2017). On Calibration of Modern Neural Networks. arXiv.