From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI
ACM Computing Surveys · 31.12.2023 · 55 (13s) · 1-42
DOI: 10.1145/3583558 ↗Kurz gefasst
Systematische Übersicht in der Zeitschrift ACM Computing Surveys, online erschienen am 13.07.2023. Verfasst von Meike Nauta und 8 weiteren Autor:innen. Die Arbeit wurde bisher 609-mal zitiert (Crossref) und nennt 307 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Ziekenhuis Groep Twente.
Abstract (Originalsprache)
The rising popularity of explainable artificial intelligence (XAI) to understand high-performing black boxes raised the question of how to evaluate explanations of machine learning (ML) models. While interpretability and explainability are often presented as a subjectively validated binary property, we consider it a multi-faceted concept. We identify 12 conceptual properties, such as Compactness and Correctness, that should be evaluated for comprehensively assessing the quality of an explanation. Our so-called Co-12 properties serve as categorization scheme for systematically reviewing the evaluation practices of more than 300 papers published in the past 7 years at major AI and ML conferences that introduce an XAI method. We find that one in three papers evaluate exclusively with anecdotal evidence, and one in five papers evaluate with users. This survey also contributes to the call for objective, quantifiable evaluation methods by presenting an extensive overview of quantitative XAI evaluation methods. Our systematic collection of evaluation methods provides researchers and practitioners with concrete tools to thoroughly validate, benchmark, and compare new and existing XAI methods. The Co-12 categorization scheme and our identified evaluation methods open up opportunities to include quantitative metrics as optimization criteria during model training to optimize for accuracy and interpretability simultaneously.
Förderung
- Ziekenhuis Groep Twente
Zitieren
APA
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., Keulen, M. V., & Seifert, C. (2023). From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI. ACM Computing Surveys, 55(13s), 1-42. https://doi.org/10.1145/3583558
Vancouver
Nauta M, Trienes J, Pathak S, Nguyen E, Peters M, Schmitt Y, et al. From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI. ACM Computing Surveys. 2023;55(13s):1-42. doi:10.1145/3583558
BibTeX
@article{Nauta2023,
author = {Nauta, Meike and Trienes, Jan and Pathak, Shreyasi and Nguyen, Elisa and Peters, Michelle and Schmitt, Yasmin and Schlötterer, Jörg and Keulen, Maurice van and Seifert, Christin},
title = {From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI},
journal = {ACM Computing Surveys},
year = {2023},
volume = {55},
number = {13s},
pages = {1--42},
doi = {10.1145/3583558}
}
EUVIMED