EUVIMEDEUVIMED entdecken
EvaluationsstudieOpen Access · CC BY 4.0

Evaluating the Appropriateness, Consistency, and Readability of ChatGPT in Critical Care Recommendations

Kaan Y Balta iD, Arshia P Javidan, Eric Walser, Robert Arntfield iD, Ross Prager

Journal of Intensive Care Medicine · 2025-02 · 40 (2) · 184-190

DOI: 10.1177/08850666241267871 ↗

Kurz gefasst

Evaluationsstudie in der Zeitschrift Journal of Intensive Care Medicine, online erschienen am 08.08.2024. Verfasst von Kaan Y Balta und 4 weiteren Autor:innen. Beteiligte Einrichtung: Schulich School of Medicine & Dentistry, Western University, London, Ontario, Canada. Hauptthemen laut MEDLINE: Critical Care, Comprehension. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

We assessed 2 versions of the large language model (LLM) ChatGPT—versions 3.5 and 4.0—in generating appropriate, consistent, and readable recommendations on core critical care topics. Research Question: How do successive large language models compare in terms of generating appropriate, consistent, and readable recommendations on core critical care topics? Design and Methods: A set of 50 LLM-generated responses to clinical questions were evaluated by 2 independent intensivists based on a 5-point Likert scale for appropriateness, consistency, and readability.

Results

ChatGPT 4.0 showed significantly higher median appropriateness scores compared to ChatGPT 3.5 (4.0 vs 3.0, P < .001). However, there was no significant difference in consistency between the 2 versions (40% vs 28%, P = 0.291). Readability, assessed by the Flesch-Kincaid Grade Level, was also not significantly different between the 2 models (14.3 vs 14.4, P = 0.93).

Interpretation

Both models produced “hallucinations”—misinformation delivered with high confidence—which highlights the risk of relying on these tools without domain expertise. Despite potential for clinical application, both models lacked consistency producing different results when asked the same question multiple times. The study underscores the need for clinicians to understand the strengths and limitations of LLMs for safe and effective implementation in critical care settings. Registration: https://osf.io/8chj7/

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Critical CareComprehensionHumansPractice Guidelines as TopicLanguage

Stichwörter der Autor:innen

appropriatenessartificial intelligencecritical carelarge language models

Autor:innen und Institutionen

  • Schulich School of Medicine & Dentistry, Western University, London, Ontario, Canada.
  • Division of Vascular Surgery, Department of Surgery, University of Toronto, Toronto, Ontario, Canada.
  • Division of Critical Care, London Health Sciences Centre, Western University, London, Ontario, Canada.
  • Department of Surgery, Trauma Program, London Health Sciences Centre, London, Ontario, Canada.

ORCID-Profile: Kaan Y Balta, Robert Arntfield

Zitieren

APA

Balta, K. Y., Javidan, A. P., Walser, E., Arntfield, R., & Prager, R. (2025). Evaluating the Appropriateness, Consistency, and Readability of ChatGPT in Critical Care Recommendations. Journal of Intensive Care Medicine, 40(2), 184-190. https://doi.org/10.1177/08850666241267871

Vancouver

Balta KY, Javidan AP, Walser E, Arntfield R, Prager R. Evaluating the Appropriateness, Consistency, and Readability of ChatGPT in Critical Care Recommendations. Journal of Intensive Care Medicine. 2025;40(2):184-190. doi:10.1177/08850666241267871

BibTeX
@article{Balta2025,
  author = {Balta, Kaan Y and Javidan, Arshia P and Walser, Eric and Arntfield, Robert and Prager, Ross},
  title = {Evaluating the Appropriateness, Consistency, and Readability of ChatGPT in Critical Care Recommendations},
  journal = {Journal of Intensive Care Medicine},
  year = {2025},
  volume = {40},
  number = {2},
  pages = {184--190},
  doi = {10.1177/08850666241267871}
}
Kostenlos starten →