Studying the explanations for the automated prediction of bug and non-bug issues using LIME and SHAP
Empirical Software Engineering · 2024-07 · 29 (4)
DOI: 10.1007/s10664-024-10469-1 ↗Kurz gefasst
Fachartikel in der Zeitschrift Empirical Software Engineering, online erschienen am 13.06.2024. Verfasst von Lukas Schulte und 2 weiteren Autor:innen. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Universität Passau.
Abstract (Originalsprache)
Context
The identification of bugs within issues reported to an issue tracking system is crucial for triage. Machine learning models have shown promising results for this task. However, we have only limited knowledge of how such models identify bugs. Explainable AI methods like LIME and SHAP can be used to increase this knowledge.
Objective
We want to understand if explainable AI provides explanations that are reasonable to us as humans and align with our assumptions about the model’s decision-making. We also want to know if the quality of predictions is correlated with the quality of explanations.
Methods
We conduct a study where we rate LIME and SHAP explanations based on their quality of explaining the outcome of an issue type prediction model. For this, we rate the quality of the explanations, i.e., if they align with our expectations and help us understand the underlying machine learning model.
Results
We found that both LIME and SHAP give reasonable explanations and that correct predictions are well explained. Further, we found that SHAP outperforms LIME due to a lower ambiguity and a higher contextuality that can be attributed to the ability of the deep SHAP variant to capture sentence fragments.
Conclusion
We conclude that the model finds explainable signals for both bugs and non-bugs. Also, we recommend that research dealing with the quality of explanations for classification tasks reports and investigates rater agreement, since the rating of explanations is highly subjective.
Förderung
- Universität Passau
Zitieren
APA
Schulte, L., Ledel, B., & Herbold, S. (2024). Studying the explanations for the automated prediction of bug and non-bug issues using LIME and SHAP. Empirical Software Engineering, 29(4). https://doi.org/10.1007/s10664-024-10469-1
Vancouver
Schulte L, Ledel B, Herbold S. Studying the explanations for the automated prediction of bug and non-bug issues using LIME and SHAP. Empirical Software Engineering. 2024;29(4). doi:10.1007/s10664-024-10469-1
BibTeX
@article{Schulte2024,
author = {Schulte, Lukas and Ledel, Benjamin and Herbold, Steffen},
title = {Studying the explanations for the automated prediction of bug and non-bug issues using LIME and SHAP},
journal = {Empirical Software Engineering},
year = {2024},
volume = {29},
number = {4},
doi = {10.1007/s10664-024-10469-1}
}
EUVIMED