The Role of Explanations in AI-Generated Alerts: Qualitative Study of Clinical Views on Explainable AI in Predictive Tools
JMIR Human Factors · 01.05.2026 · 13 · e81460-e81460
DOI: 10.2196/81460 ↗Kurz gefasst
Qualitative Studie in der Zeitschrift JMIR Human Factors, online erschienen am 01.05.2026. Verfasst von Jessica Rahman und 6 weiteren Autor:innen. Beteiligte Einrichtung: Australian e-Health Research Centre, Commonwealth Scientific and Industrial Research Organisation (CSIRO), 160 Hawkesbury Road, Sydney, New South Wal…. Hauptthemen laut MEDLINE: Artificial Intelligence, Decision Support Systems, Clinical. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 77 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Artificial intelligence (AI)–driven clinical decision support (CDS) tools offer promising solutions for health care delivery by optimizing resource allocation, detecting deterioration, and enabling early interventions. However, adoption remains limited due to insufficient validation and a lack of transparency and trust. Explainable AI (XAI) seeks to improve user understanding of AI outputs; however, how clinicians interpret and integrate these explanations into their decision-making remains underexplored. Furthermore, discrepancies in explanations, known as the “disagreement problem,” can undermine trust and, at worst, lead to poor clinical decisions.
Objective
This study examines clinicians’ perspectives on the role and value of explainability in AI-driven CDS tools within Australian critical care settings and the impact of discrepancies in AI-generated explanations on clinical decision-making.
Methods
Qualitative data were collected using semistructured interviews with 14 clinical experts, incorporating scenario-based exercises, and were analyzed using inductive thematic analysis.
Results
Clinicians valued explainability, particularly in complex or unfamiliar situations, when explanations were clear, plausible, and actionable. Trust and perceived usefulness extended beyond explanation quality, encompassing factors such as system accuracy, alignment with clinicians’ reasoning, workflow integration, and perceived reliability. Discrepancies in explanations generated by different XAI methods were not a major concern, provided that the AI-generated predictive alerts were accurate.
Conclusions
This study provides design recommendations for developing trustworthy, user-centric CDS tools that incorporate XAI. Findings highlight that explainability is critical for establishing initial trust in AI-driven tools by supporting perceived usefulness, but its importance diminishes over time and with user expertise and familiarity, as learned usefulness takes precedence. Recommendations highlight the importance of aligning the design and implementation of AI tools with clinicians’ needs to enhance trust, mitigate risks, and promote successful adoption for improved patient outcomes.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Australian e-Health Research Centre, Commonwealth Scientific and Industrial Research Organisation (CSIRO), 160 Hawkesbury Road, Sydney, New South Wales, 2145, Australia, 61 2 9325 3016.
- Australian e-Health Research Centre, Commonwealth Scientific and Industrial Research Organisation (CSIRO), Brisbane, Queensland, Australia.
- School of Nursing and Midwifery, Griffith University, Brisbane, Australia.
- Australian Institute of Health Innovation, Macquarie University, Sydney, Australia.
- Intensive Care Unit, Princess Alexandra Hospital, Brisbane, Australia.
ORCID-Profile: Jessica Rahman, Alana Delaforce, DanaKai Bradford, Jane Li, Farah Magrabi, David Cook, Aida Brankovic
Zitieren
APA
Rahman, J., Delaforce, A., Bradford, D., Li, J., Magrabi, F., Cook, D., & Brankovic, A. (2026). The Role of Explanations in AI-Generated Alerts: Qualitative Study of Clinical Views on Explainable AI in Predictive Tools. JMIR Human Factors, 13, e81460-e81460. https://doi.org/10.2196/81460
Vancouver
Rahman J, Delaforce A, Bradford D, Li J, Magrabi F, Cook D, et al. The Role of Explanations in AI-Generated Alerts: Qualitative Study of Clinical Views on Explainable AI in Predictive Tools. JMIR Human Factors. 2026;13:e81460-e81460. doi:10.2196/81460
BibTeX
@article{Rahman2026,
author = {Rahman, Jessica and Delaforce, Alana and Bradford, DanaKai and Li, Jane and Magrabi, Farah and Cook, David and Brankovic, Aida},
title = {The Role of Explanations in AI-Generated Alerts: Qualitative Study of Clinical Views on Explainable AI in Predictive Tools},
journal = {JMIR Human Factors},
year = {2026},
volume = {13},
pages = {e81460--e81460},
doi = {10.2196/81460}
}
EUVIMED