XAI Systems Evaluation: A Review of Human and Computer-Centred Methods
Source abstract
Abstract
The lack of transparency of powerful Machine Learning systems paired with their growth in popularity over the last decade led to the emergence of the eXplainable Artificial Intelligence (XAI) field. Instead of focusing solely on obtaining highly performing models, researchers also develop explanation techniques that help better understand the system’s reasoning for a particular output. An explainable system can be designed, developed, and evaluated from different perspectives, which enables researchers from different disciplines to work together on this topic. However, the multidisciplinary nature of XAI systems creates new challenges for condensing and structuring adequate methodologies to design and evaluate such systems. This paper presents a survey of Human-centred and Computer-centred methods to evaluate XAI systems. We propose a new taxonomy to categorize XAI evaluation methods more clearly and intuitively. This categorization gathers knowledge from different disciplines and organizes the evaluation methods according to a set of categories that represent key properties of XAI systems. Possible ways to use the proposed taxonomy in the design and evaluation of XAI systems are also discussed, alongside with some concluding remarks and future directions of research.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Explainable Artificial Intelligence (XAI) · Bayesian Modeling and Causal Inference · Scientific Computing and Data Management
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Fraunhofer Portugal AICOS, Rua Alfredo Allen 455/461, 4200-135 Porto, Portugal
First Solutions—Sistemas de Informação S.A., 4450-102 Matosinhos, Portugal
Funding
Transparent Artificial Medical Intelligence (TAMI) · Operational Programme for Competitiveness and Internationalization (COMPETE 2020) · Fundação para a Ciência and Technology (FCT) · Carnegie Mellon University · European Regional Development Fund
References (77)
Show source-provided references
1. Notions of explainability and evaluation approaches for explainable artificial intelligence10.1016/j.inffus.2021.05.009 ↗
2. (2022, May 15). General Data Protection Regulation (GDPR)–Official Legal Text. Available online: https://gdpr-info.eu/.
3. A multidisciplinary survey and framework for design and evaluation of explainable AI systems10.1145/3387166 ↗
4. Explanation in Artificial Intelligence: Insights from the Social Sciences10.1016/j.artint.2018.07.007 ↗
5. Liao, Q.V., and Varshney, K.R. (2022). Human-Centered Explainable AI (XAI): From Algorithms to User Experiences. arXiv.
6. The role of explainability in creating trustworthy artificial intelligence for health care: A comprehensive survey of the terminology, design choices, and evaluation strategies10.1016/j.jbi.2020.103655 ↗
7. Eiband, M., Buschek, D., Kremer, A., and Hussmann, H. (2019). The Impact of Placebic Explanations on Trust in Intelligent Systems. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, ACM.10.1145/3290607.3312787 ↗
8. Herman, B. (2017). The promise and peril of human evaluation for model interpretability. arXiv.
9. Doshi-Velez, F., and Kim, B. (2017). Towards a rigorous science of interpretable machine learning. arXiv.
10. Zhou, J., Gandomi, A.H., Chen, F., and Holzinger, A. (2021). Evaluating the quality of machine learning explanations: A survey on methods and metrics. Electronics, 10.10.3390/electronics10050593 ↗
11. Evaluating XAI: A Comparison of Rule-Based and Example-Based Explanations10.1016/j.artint.2020.103404 ↗
12. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)10.1109/ACCESS.2018.2870052 ↗
13. Hedstrom, A., Weber, L., Bareeva, D., Motzkus, F., Samek, W., Lapuschkin, S., and Hohne, M.M.C. (2022). Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations. arXiv.
14. Kahneman, D. (2011). Thinking, Fast and Slow, Macmillan.
15. DARPA’s Explainable Artificial Intelligence (XAI) Program
16. Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J.M.F., and Eckersley, P. (2020). Explainable Machine Learning in Deployment. arXiv.10.1145/3351095.3375624 ↗
17. Bussone, A., Stumpf, S., and O’Sullivan, D. (2015, January 21–23). The Role of Explanations on Trust and Reliance in Clinical Decision Support Systems. Proceedings of the 2015 International Conference on Healthcare Informatics, Dallas, TX, USA.10.1109/ICHI.2015.26 ↗
18. Does Projection into Use Improve Trust and Exploration? An Example with a Cruise Control System10.1016/j.ssci.2009.03.015 ↗
19. Berkovsky, S., Taib, R., and Conway, D. (2017, January 13–16). How to Recommend? User Trust Factors in Movie Recommender Systems. Proceedings of the 22nd International Conference on Intelligent User Interfaces, Limassol, Cyprus.10.1145/3025171.3025209 ↗
20. Nourani, M., Kabir, S., Mohseni, S., and Ragan, E.D. (2019, January 28–30). The Effects of Meaningful and Meaningless Explanations on Trust and Perceived System Accuracy in Intelligent Systems. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Washington, DC, USA.10.1609/hcomp.v7i1.5284 ↗
21. Yin, M., Wortman Vaughan, J., and Wallach, H. (2019, January 4–9). Understanding the Effect of Accuracy on Trust in Machine Learning Models. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK.10.1145/3290605.3300509 ↗
22. Kunkel, J., Donkers, T., Michael, L., Barbu, C.M., and Ziegler, J. (2019, January 4–9). Let Me Explain: Impact of Personal and Impersonal Explanations on Trust in Recommender Systems. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK.10.1145/3290605.3300717 ↗
23. Holliday, D., Wilson, S., and Stumpf, S. (2016, January 7–10). User Trust in Intelligent Systems: A Journey Over Time. Proceedings of the 21st International Conference on Intelligent User Interfaces, Sonoma, CA, USA.10.1145/2856767.2856811 ↗
24. Yang, X.J., Unhelkar, V.V., Li, K., and Shah, J.A. (2017, January 6–9). Evaluating Effects of User Experience and System Transparency on Trust in Automation. Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, Vienna, Austria.10.1145/2909824.3020230 ↗
25. Zhang, Y., Liao, Q.V., and Bellamy, R.K.E. (2020, January 27–30). Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-assisted Decision Making. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, Barcelona, Spain.10.1145/3351095.3372852 ↗
26. A Framework for Considering Comprehensibility in Modeling10.1089/big.2016.0007 ↗
27. Madsen, M., and Gregor, S. (2000, January 6–8). Measuring Human-Computer Trust. Proceedings of the 11th Australasian Conference on Information Systems, Brisbane, Australia.
28. Rader, E., and Gray, R. (2015, January 18–23). Understanding User Beliefs About Algorithmic Curation in the Facebook News Feed. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Korea.10.1145/2702123.2702174 ↗
29. Lim, B.Y., and Dey, A.K. (October, January 30). Assessing Demand for Intelligibility in Context-Aware Applications. Proceedings of the 11th International Conference on Ubiquitous Computing, Orlando, FL, USA.
30. Nothdurft, F., Richter, F., and Minker, W. (2014, January 18–20). Probabilistic Human-Computer Trust Handling. Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), Philadelphia, PA, USA.10.3115/v1/W14-4307 ↗
31. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13–17). “ Why should i trust you?” Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939778 ↗
32. Ribeiro, M.T., Singh, S., and Guestrin, C. (2018, January 2–7). Anchors: High Precision Model-Agnostic Explanations. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11491 ↗
33. Bansal, G., Nushi, B., Kamar, E., Weld, D.S., Lasecki, W.S., and Horvitz, E. (February, January 27). Updates in Human-AI Teams: Understanding and Addressing the Performance/Compatibility Tradeoff. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
34. Bansal, G., Nushi, B., Kamar, E., Lasecki, W., Weld, D.S., and Horvitz, E. (2019, January 28). Beyond Accuracy: The Role of Mental Models in Human-AI Team Performance. Proceedings of the Seventh AAAI Conference on Human Computation and Crowdsourcing, Washington, DC, USA.10.1609/hcomp.v7i1.5285 ↗
35. Nushi, B., Kamar, E., and Horvitz, E. (2018). Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure. arXiv.10.1609/hcomp.v6i1.13337 ↗
36. Shen, H., and Huang, T.H.K. (2020). How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labels. arXiv.10.1609/hcomp.v8i1.7477 ↗
37. Kulesza, T., Stumpf, S., Burnett, M., Wong, W.K., Riche, Y., Moore, T., Oberst, I., Shinsel, A., and McIntosh, K. (2010, January 21–25). Explanatory Debugging: Supporting End-User Debugging of Machine-Learned Programs. Proceedings of the 2010 IEEE Symposium on Visual Languages and Human-Centric Computing, Leganes, Spain.10.1109/VLHCC.2010.15 ↗
38. Binns, R., Van Kleek, M., Veale, M., Lyngs, U., Zhao, J., and Shadbolt, N. (2018, January 21–26). ’It’s Reducing a Human Being to a Percentage’; Perceptions of Justice in Algorithmic Decisions. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.10.1145/3173574.3173951 ↗
39. Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., and Sayres, R. (2018, January 10–15). Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden.
40. Lakkaraju, H., Bach, S.H., and Leskovec, J. (2016, January 13–17). Interpretable decision sets: A joint framework for description and prediction. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA.10.1145/2939672.2939874 ↗
41. How Should I Explain? A Comparison of Different Explanation Types for Recommender Systems10.1016/j.ijhcs.2013.12.007 ↗
42. Lim, B.Y., Dey, A.K., and Avrahami, D. (2009, January 4–9). Why and Why Not Explanations Improve the Intelligibility of Context-Aware Intelligent Systems. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA.10.1145/1518701.1519023 ↗
43. Kahng, M., Andrews, P.Y., Kalro, A., and Chau, D.H. (2017). ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models. arXiv.10.1109/TVCG.2017.2744718 ↗
44. Strobelt, H., Gehrmann, S., Pfister, H., and Rush, A.M. (2017). LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks. arXiv.10.1109/TVCG.2017.2744158 ↗
45. Coppers, S., Van den Bergh, J., Luyten, K., Coninx, K., van der Lek-Ciudin, I., Vanallemeersch, T., and Vandeghinste, V. (2018, January 21–26). Intellingo: An Intelligible Translation Environment. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, Montreal, QC, Canada.10.1145/3173574.3174098 ↗
46. Poursabzi-Sangdeh, F., Goldstein, D.G., Hofman, J.M., Wortman Vaughan, J.W., and Wallach, H. (2021, January 8–13). Manipulating and Measuring Model Interpretability. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Yokohama, Japan.10.1145/3411764.3445315 ↗
47. Bunt, A., Lount, M., and Lauzon, C. (2012, January 14–17). Are Explanations Always Important?: A Study of Deployed, Low-Cost Intelligent Interactive Systems. Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces, Lisbon, Portugal.10.1145/2166966.2166996 ↗
48. Samuel, S.Z.S., Kamakshi, V., Lodhi, N., and Krishnan, N.C. (2021). Evaluation of Saliency-based Explainability Method. arXiv.
49. Interpretability in healthcare: A comparative study of local machine learning interpretability techniques10.1111/coin.12410 ↗
50. Honegger, M. (2018). Shedding light on black box machine learning algorithms: Development of an axiomatic framework to assess the quality of methods that explain individual predictions. arXiv.
51. Nguyen, A., and Martínez, M. (2020). On Quantitative Aspects of Model Interpretability. arXiv.
52. Slack, D., Friedler, S.A., Scheidegger, C., and Roy, C.D. (2019). Assessing the local interpretability of machine learning models. arXiv.
53. Hara, S., and Hayashi, K. (2016). Making tree ensembles interpretable. arXiv.
54. Interpreting tree ensembles with intrees10.1007/s41060-018-0144-8 ↗
55. Lakkaraju, H., Kamar, E., Caruana, R., and Leskovec, J. (2017). Interpretable & explorable approximations of black box models. arXiv.
56. Bhatt, U., Weller, A., and Moura, J.M. (2020). Evaluating and aggregating feature-based model explanations. arXiv.10.24963/ijcai.2020/417 ↗
57. Bau, D., Zhou, B., Khosla, A., Oliva, A., and Torralba, A. (2017, January 21–26). Network dissection: Quantifying interpretability of deep visual representations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.354 ↗
58. Zhang, Q., Wu, Y.N., and Zhu, S.C. (2018, January 18–23). Interpretable convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00920 ↗
59. Zhang, Q., Cao, R., Shi, F., Wu, Y.N., and Zhu, S.C. (2018, January 2–7). Interpreting cnn knowledge via an explanatory graph. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11819 ↗
60. Laugel, T., Lesot, M.J., Marsala, C., Renard, X., and Detyniecki, M. (2019). The dangers of post-hoc interpretability: Unjustified counterfactual explanations. arXiv.10.24963/ijcai.2019/388 ↗
61. Plumb, G., Al-Shedivat, M., Cabrera, A.A., Perer, A., Xing, E., and Talwalkar, A. (2019). Regularizing black-box models for improved interpretability. arXiv.
62. Towards robust interpretability with self-explaining neural networks
63. Alvarez-Melis, D., and Jaakkola, T.S. (2018). On the robustness of interpretability methods. arXiv.
64. Sundararajan, M., Taly, A., and Yan, Q. (2017, January 6–11). Axiomatic attribution for deep networks. Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia.
65. Methods for interpreting and understanding deep neural networks10.1016/j.dsp.2017.10.011 ↗
66. Kindermans, P.J., Hooker, S., Adebayo, J., Alber, M., Schütt, K.T., Dähne, S., Erhan, D., and Kim, B. (2019). The (un) reliability of saliency methods. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer.10.1007/978-3-030-28954-6_14 ↗
67. Dissecting explanatory power10.1007/s11098-008-9324-z ↗
68. On the (in) fidelity and sensitivity of explanations
69. Deng, H., Zou, N., Du, M., Chen, W., Feng, G., and Hu, X. (2021, January 2–9). A Unified Taylor Framework for Revisiting Attribution Methods. Proceedings of the AAAI Conference on Artificial Intelligence, Vancouver, DC, USA.10.1609/aaai.v35i13.17365 ↗
70. Kohlbrenner, M., Bauer, A., Nakajima, S., Binder, A., Samek, W., and Lapuschkin, S. (2020, January 19–24). Towards best practice in explaining neural network decisions with LRP. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.10.1109/IJCNN48605.2020.9206975 ↗
71. Hooker, S., Erhan, D., Kindermans, P.J., and Kim, B. (2018). A benchmark for interpretability methods in deep neural networks. arXiv.
72. Evaluating the visualization of what a deep neural network has learned10.1109/TNNLS.2016.2599820 ↗
73. Sanity Checks for Saliency Maps
74. Ignatiev, A. (2021, January 7–15). Towards Trustable Explainable AI. Proceedings of the 29th International Joint Conference on Artificial Intelligence, Yokohama, Japan.10.24963/ijcai.2020/726 ↗
75. The Mindlessness of Ostensibly Thoughtful Action: The Role of “Placebic” Information in Interpersonal Interaction10.1037/0022-3514.36.6.635 ↗
76. Buçinca, Z., Lin, P., Gajos, K.Z., and Glassman, E.L. (2020, January 17–20). Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems. Proceedings of the 25th International Conference on Intelligent User Interfaces, Cagliari, Italy.10.1145/3377325.3377498 ↗
77. Affective analytics of demonstration sites10.1016/j.engappai.2019.03.001 ↗