EUVIMEDOpdag EUVIMED
Qualitative StudieOpen access

The Role of Explanations in AI-Generated Alerts: Qualitative Study of Clinical Views on Explainable AI in Predictive Tools

Jessica Rahman iD, Alana Delaforce iD, DanaKai Bradford iD, Jane Li iD, Farah Magrabi iD, David Cook iD, Aida Brankovic iD

JMIR Human Factors · 2026-05-01 · 13 · e81460-e81460

DOI: 10.2196/81460 ↗

At a glance

Qualitative Studie in JMIR Human Factors, published online on 2026-05-01. Written by Jessica Rahman and 6 co-authors. Institution: Australian e-Health Research Centre, Commonwealth Scientific and Industrial Research Organisation (CSIRO), 160 Hawkesbury Road, Sydney, New South Wal…. Major MEDLINE topics: Artificial Intelligence, Decision Support Systems, Clinical. Cited 3 times so far (Crossref); 77 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Artificial intelligence (AI)–driven clinical decision support (CDS) tools offer promising solutions for health care delivery by optimizing resource allocation, detecting deterioration, and enabling early interventions. However, adoption remains limited due to insufficient validation and a lack of transparency and trust. Explainable AI (XAI) seeks to improve user understanding of AI outputs; however, how clinicians interpret and integrate these explanations into their decision-making remains underexplored. Furthermore, discrepancies in explanations, known as the “disagreement problem,” can undermine trust and, at worst, lead to poor clinical decisions.

Objective

This study examines clinicians’ perspectives on the role and value of explainability in AI-driven CDS tools within Australian critical care settings and the impact of discrepancies in AI-generated explanations on clinical decision-making.

Methods

Qualitative data were collected using semistructured interviews with 14 clinical experts, incorporating scenario-based exercises, and were analyzed using inductive thematic analysis.

Results

Clinicians valued explainability, particularly in complex or unfamiliar situations, when explanations were clear, plausible, and actionable. Trust and perceived usefulness extended beyond explanation quality, encompassing factors such as system accuracy, alignment with clinicians’ reasoning, workflow integration, and perceived reliability. Discrepancies in explanations generated by different XAI methods were not a major concern, provided that the AI-generated predictive alerts were accurate.

Conclusions

This study provides design recommendations for developing trustworthy, user-centric CDS tools that incorporate XAI. Findings highlight that explainability is critical for establishing initial trust in AI-driven tools by supporting perceived usefulness, but its importance diminishes over time and with user expertise and familiarity, as learned usefulness takes precedence. Recommendations highlight the importance of aligning the design and implementation of AI tools with clinicians’ needs to enhance trust, mitigate risks, and promote successful adoption for improved patient outcomes.

Subjects

MeSH terms (MEDLINE)

Artificial IntelligenceDecision Support Systems, ClinicalAustraliaQualitative ResearchInterviews as TopicPhysiciansHumansAttitude of Health PersonnelEarly Warning ScoreMaleFemale

Author keywords

artificial intelligenceclinical decision supportdisagreement in XAIexplainable AIexplainable artificial intelligencehuman factorsqualitative research

Authors and institutions

  • Australian e-Health Research Centre, Commonwealth Scientific and Industrial Research Organisation (CSIRO), 160 Hawkesbury Road, Sydney, New South Wales, 2145, Australia, 61 2 9325 3016.
  • Australian e-Health Research Centre, Commonwealth Scientific and Industrial Research Organisation (CSIRO), Brisbane, Queensland, Australia.
  • School of Nursing and Midwifery, Griffith University, Brisbane, Australia.
  • Australian Institute of Health Innovation, Macquarie University, Sydney, Australia.
  • Intensive Care Unit, Princess Alexandra Hospital, Brisbane, Australia.

ORCID profiles: Jessica Rahman, Alana Delaforce, DanaKai Bradford, Jane Li, Farah Magrabi, David Cook, Aida Brankovic

Cite

APA

Rahman, J., Delaforce, A., Bradford, D., Li, J., Magrabi, F., Cook, D., & Brankovic, A. (2026). The Role of Explanations in AI-Generated Alerts: Qualitative Study of Clinical Views on Explainable AI in Predictive Tools. JMIR Human Factors, 13, e81460-e81460. https://doi.org/10.2196/81460

Vancouver

Rahman J, Delaforce A, Bradford D, Li J, Magrabi F, Cook D, et al. The Role of Explanations in AI-Generated Alerts: Qualitative Study of Clinical Views on Explainable AI in Predictive Tools. JMIR Human Factors. 2026;13:e81460-e81460. doi:10.2196/81460

BibTeX
@article{Rahman2026,
  author = {Rahman, Jessica and Delaforce, Alana and Bradford, DanaKai and Li, Jane and Magrabi, Farah and Cook, David and Brankovic, Aida},
  title = {The Role of Explanations in AI-Generated Alerts: Qualitative Study of Clinical Views on Explainable AI in Predictive Tools},
  journal = {JMIR Human Factors},
  year = {2026},
  volume = {13},
  pages = {e81460--e81460},
  doi = {10.2196/81460}
}
Start for free →