EUVIMEDEUVIMED entdecken
Qualitative StudieOpen Access

The Role of Explanations in AI-Generated Alerts: Qualitative Study of Clinical Views on Explainable AI in Predictive Tools

Jessica Rahman iD, Alana Delaforce iD, DanaKai Bradford iD, Jane Li iD, Farah Magrabi iD, David Cook iD, Aida Brankovic iD

JMIR Human Factors · 01.05.2026 · 13 · e81460-e81460

DOI: 10.2196/81460 ↗

Kurz gefasst

Qualitative Studie in der Zeitschrift JMIR Human Factors, online erschienen am 01.05.2026. Verfasst von Jessica Rahman und 6 weiteren Autor:innen. Beteiligte Einrichtung: Australian e-Health Research Centre, Commonwealth Scientific and Industrial Research Organisation (CSIRO), 160 Hawkesbury Road, Sydney, New South Wal…. Hauptthemen laut MEDLINE: Artificial Intelligence, Decision Support Systems, Clinical. Die Arbeit wurde bisher 3-mal zitiert (Crossref) und nennt 77 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Artificial intelligence (AI)–driven clinical decision support (CDS) tools offer promising solutions for health care delivery by optimizing resource allocation, detecting deterioration, and enabling early interventions. However, adoption remains limited due to insufficient validation and a lack of transparency and trust. Explainable AI (XAI) seeks to improve user understanding of AI outputs; however, how clinicians interpret and integrate these explanations into their decision-making remains underexplored. Furthermore, discrepancies in explanations, known as the “disagreement problem,” can undermine trust and, at worst, lead to poor clinical decisions.

Objective

This study examines clinicians’ perspectives on the role and value of explainability in AI-driven CDS tools within Australian critical care settings and the impact of discrepancies in AI-generated explanations on clinical decision-making.

Methods

Qualitative data were collected using semistructured interviews with 14 clinical experts, incorporating scenario-based exercises, and were analyzed using inductive thematic analysis.

Results

Clinicians valued explainability, particularly in complex or unfamiliar situations, when explanations were clear, plausible, and actionable. Trust and perceived usefulness extended beyond explanation quality, encompassing factors such as system accuracy, alignment with clinicians’ reasoning, workflow integration, and perceived reliability. Discrepancies in explanations generated by different XAI methods were not a major concern, provided that the AI-generated predictive alerts were accurate.

Conclusions

This study provides design recommendations for developing trustworthy, user-centric CDS tools that incorporate XAI. Findings highlight that explainability is critical for establishing initial trust in AI-driven tools by supporting perceived usefulness, but its importance diminishes over time and with user expertise and familiarity, as learned usefulness takes precedence. Recommendations highlight the importance of aligning the design and implementation of AI tools with clinicians’ needs to enhance trust, mitigate risks, and promote successful adoption for improved patient outcomes.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceDecision Support Systems, ClinicalAustraliaQualitative ResearchInterviews as TopicPhysiciansHumansAttitude of Health PersonnelEarly Warning ScoreMaleFemale

Stichwörter der Autor:innen

artificial intelligenceclinical decision supportdisagreement in XAIexplainable AIexplainable artificial intelligencehuman factorsqualitative research

Autor:innen und Institutionen

  • Australian e-Health Research Centre, Commonwealth Scientific and Industrial Research Organisation (CSIRO), 160 Hawkesbury Road, Sydney, New South Wales, 2145, Australia, 61 2 9325 3016.
  • Australian e-Health Research Centre, Commonwealth Scientific and Industrial Research Organisation (CSIRO), Brisbane, Queensland, Australia.
  • School of Nursing and Midwifery, Griffith University, Brisbane, Australia.
  • Australian Institute of Health Innovation, Macquarie University, Sydney, Australia.
  • Intensive Care Unit, Princess Alexandra Hospital, Brisbane, Australia.

ORCID-Profile: Jessica Rahman, Alana Delaforce, DanaKai Bradford, Jane Li, Farah Magrabi, David Cook, Aida Brankovic

Zitieren

APA

Rahman, J., Delaforce, A., Bradford, D., Li, J., Magrabi, F., Cook, D., & Brankovic, A. (2026). The Role of Explanations in AI-Generated Alerts: Qualitative Study of Clinical Views on Explainable AI in Predictive Tools. JMIR Human Factors, 13, e81460-e81460. https://doi.org/10.2196/81460

Vancouver

Rahman J, Delaforce A, Bradford D, Li J, Magrabi F, Cook D, et al. The Role of Explanations in AI-Generated Alerts: Qualitative Study of Clinical Views on Explainable AI in Predictive Tools. JMIR Human Factors. 2026;13:e81460-e81460. doi:10.2196/81460

BibTeX
@article{Rahman2026,
  author = {Rahman, Jessica and Delaforce, Alana and Bradford, DanaKai and Li, Jane and Magrabi, Farah and Cook, David and Brankovic, Aida},
  title = {The Role of Explanations in AI-Generated Alerts: Qualitative Study of Clinical Views on Explainable AI in Predictive Tools},
  journal = {JMIR Human Factors},
  year = {2026},
  volume = {13},
  pages = {e81460--e81460},
  doi = {10.2196/81460}
}
Kostenlos starten →