EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Studying the explanations for the automated prediction of bug and non-bug issues using LIME and SHAP

Lukas Schulte, Benjamin Ledel, Steffen Herbold

Empirical Software Engineering · 2024-07 · 29 (4)

DOI: 10.1007/s10664-024-10469-1 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Empirical Software Engineering, online erschienen am 13.06.2024. Verfasst von Lukas Schulte und 2 weiteren Autor:innen. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Universität Passau.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Context

The identification of bugs within issues reported to an issue tracking system is crucial for triage. Machine learning models have shown promising results for this task. However, we have only limited knowledge of how such models identify bugs. Explainable AI methods like LIME and SHAP can be used to increase this knowledge.

Objective

We want to understand if explainable AI provides explanations that are reasonable to us as humans and align with our assumptions about the model’s decision-making. We also want to know if the quality of predictions is correlated with the quality of explanations.

Methods

We conduct a study where we rate LIME and SHAP explanations based on their quality of explaining the outcome of an issue type prediction model. For this, we rate the quality of the explanations, i.e., if they align with our expectations and help us understand the underlying machine learning model.

Results

We found that both LIME and SHAP give reasonable explanations and that correct predictions are well explained. Further, we found that SHAP outperforms LIME due to a lower ambiguity and a higher contextuality that can be attributed to the ability of the deep SHAP variant to capture sentence fragments.

Conclusion

We conclude that the model finds explainable signals for both bugs and non-bugs. Also, we recommend that research dealing with the quality of explanations for classification tasks reports and investigates rater agreement, since the rating of explanations is highly subjective.

Förderung

  • Universität Passau

Zitieren

APA

Schulte, L., Ledel, B., & Herbold, S. (2024). Studying the explanations for the automated prediction of bug and non-bug issues using LIME and SHAP. Empirical Software Engineering, 29(4). https://doi.org/10.1007/s10664-024-10469-1

Vancouver

Schulte L, Ledel B, Herbold S. Studying the explanations for the automated prediction of bug and non-bug issues using LIME and SHAP. Empirical Software Engineering. 2024;29(4). doi:10.1007/s10664-024-10469-1

BibTeX
@article{Schulte2024,
  author = {Schulte, Lukas and Ledel, Benjamin and Herbold, Steffen},
  title = {Studying the explanations for the automated prediction of bug and non-bug issues using LIME and SHAP},
  journal = {Empirical Software Engineering},
  year = {2024},
  volume = {29},
  number = {4},
  doi = {10.1007/s10664-024-10469-1}
}
Kostenlos starten →