EUVIMEDObjevte EUVIMED
ÜbersichtsarbeitSystematische ÜbersichtOpen access

How Explainable Artificial Intelligence Can Increase or Decrease Clinicians’ Trust in AI Applications in Health Care: Systematic Review

Wie Explainable Artificial Intelligence das Vertrauen von Kliniker:innen in KI-Anwendungen im Gesundheitswesen erhöhen oder senken kann: systematische Übersichtsarbeit

Rikard Rosenbacke iD, Åsa Melhus iD, Martin McKee iD, David Stuckler iD

JMIR AI · 2024-10-30 · 3 · e53207

DOI: 10.2196/53207 ↗

Auf einen Blick

Die Übersichtsarbeit zu 10 Studien zeigt: Explainable AI (XAI) kann das Vertrauen von Kliniker:innen in KI erhöhen, aber auch senken. Entscheidend war die Qualität der Erklärungen.

Fragestellung

Welchen Einfluss hat Explainable AI auf das Vertrauen von Kliniker:innen in KI-gestützte klinische Entscheidungen?

Methode

Systematische Übersichtsarbeit nach PRISMA; Suche in PubMed und Web of Science. Eingeschlossen wurden Studien, die den Einfluss von XAI auf Vertrauen empirisch maßen.

Ergebnisse

10 Studien erfüllten die Einschlusskriterien. 5 berichteten mehr Vertrauen mit XAI, 3 fanden keinen signifikanten Effekt, 2 zeigten je nach Komplexität und Kohärenz Zu- oder Abnahme des Vertrauens.

Einschränkungen

Verzerrungsrisiko in allen Studien moderat bis moderat-hoch; Vertrauen vor allem kognitiv operationalisiert, nur 2 Studien nutzten auch affektbasierte Maße.

Schlussfolgerung der Autor:innen

Zu viel Vertrauen in falsche KI-Empfehlungen kann die klinische Genauigkeit beeinträchtigen, ebenso wie Misstrauen gegenüber korrekten Empfehlungen. XAI sollte so gestaltet werden, dass ein angemessenes Vertrauensniveau entsteht.

Kernaussagen

  • XAI erhöhte in 5 Studien das Vertrauen von Kliniker:innen
  • 3 Studien zeigten keinen signifikanten Effekt auf Vertrauen
  • Erklärungsqualität kann Vertrauen erhöhen oder mindern

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Übersichtsarbeit in JMIR AI, published online on 2024-10-30. Written by Rikard Rosenbacke and 3 co-authors. Institution: Centre for Corporate Governance, Department of Accounting, Copenhagen Business School, Frederiksberg, Denmark. Cited 346 times so far (Crossref); 42 references. The full text is freely available in PubMed Central.

Auf einen BlickAbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Artificial intelligence (AI) has significant potential in clinical practice. However, its “black box” nature can lead clinicians to question its value. The challenge is to create sufficient trust for clinicians to feel comfortable using AI, but not so much that they defer to it even when it produces results that conflict with their clinical judgment in ways that lead to incorrect decisions. Explainable AI (XAI) aims to address this by providing explanations of how AI algorithms reach their conclusions. However, it remains unclear whether such explanations foster an appropriate degree of trust to ensure the optimal use of AI in clinical practice.

Objective

This study aims to systematically review and synthesize empirical evidence on the impact of XAI on clinicians’ trust in AI-driven clinical decision-making.

Methods

A systematic review was conducted in accordance with PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, searching PubMed and Web of Science databases. Studies were included if they empirically measured the impact of XAI on clinicians’ trust using cognition- or affect-based measures. Out of 778 articles screened, 10 met the inclusion criteria. We assessed the risk of bias using standard tools appropriate to the methodology of each paper.

Results

The risk of bias in all papers was moderate or moderate to high. All included studies operationalized trust primarily through cognitive-based definitions, with 2 also incorporating affect-based measures. Out of these, 5 studies reported that XAI increased clinicians’ trust compared with standard AI, particularly when the explanations were clear, concise, and relevant to clinical practice. In addition, 3 studies found no significant effect of XAI on trust, and the presence of explanations does not automatically improve trust. Notably, 2 studies highlighted that XAI could either enhance or diminish trust, depending on the complexity and coherence of the provided explanations. The majority of studies suggest that XAI has the potential to enhance clinicians’ trust in recommendations generated by AI. However, complex or contradictory explanations can undermine this trust. More critically, trust in AI is not inherently beneficial, as AI recommendations are not infallible. These findings underscore the nuanced role of explanation quality and suggest that trust can be modulated through the careful design of XAI systems.

Conclusions

Excessive trust in incorrect advice generated by AI can adversely impact clinical accuracy, just as can happen when correct advice is distrusted. Future research should focus on refining both cognitive and affect-based measures of trust and on developing strategies to achieve an appropriate balance in terms of trust, preventing both blind trust and undue skepticism. Optimizing trust in AI systems is essential for their effective integration into clinical practice.

Subjects

Author keywords

XAIaffect-based measuresclinical decision-makingclinical informaticsclinical useclinician trustcognitive measuresexplainable artificial intelligencetrustworthy AI

Authors and institutions

  • Centre for Corporate Governance, Department of Accounting, Copenhagen Business School, Frederiksberg, Denmark.
  • Department of Medical Sciences, Clinical Microbiology, Uppsala University, Uppsala, Sweden.
  • European Observatory on Health Systems and Policies, London School of Hygiene & Tropical Medicine, London, United Kingdom.
  • Department of Social and Political Sciences, Bocconi University, Milano, Italy.

ORCID profiles: Rikard Rosenbacke, Åsa Melhus, Martin McKee, David Stuckler

Cite

APA

Rosenbacke, R., Melhus, Å., McKee, M., & Stuckler, D. (2024). How Explainable Artificial Intelligence Can Increase or Decrease Clinicians’ Trust in AI Applications in Health Care: Systematic Review. JMIR AI, 3, e53207. https://doi.org/10.2196/53207

Vancouver

Rosenbacke R, Melhus Å, McKee M, Stuckler D. How Explainable Artificial Intelligence Can Increase or Decrease Clinicians’ Trust in AI Applications in Health Care: Systematic Review. JMIR AI. 2024;3:e53207. doi:10.2196/53207

BibTeX
@article{Rosenbacke2024,
  author = {Rosenbacke, Rikard and Melhus, Åsa and McKee, Martin and Stuckler, David},
  title = {How Explainable Artificial Intelligence Can Increase or Decrease Clinicians’ Trust in AI Applications in Health Care: Systematic Review},
  journal = {JMIR AI},
  year = {2024},
  volume = {3},
  pages = {e53207},
  doi = {10.2196/53207}
}
Start for free →