Evaluating the Appropriateness, Consistency, and Readability of ChatGPT in Critical Care Recommendations
Journal of Intensive Care Medicine · 2025-02 · 40 (2) · 184-190
DOI: 10.1177/08850666241267871 ↗Kurz gefasst
Evaluationsstudie in der Zeitschrift Journal of Intensive Care Medicine, online erschienen am 08.08.2024. Verfasst von Kaan Y Balta und 4 weiteren Autor:innen. Beteiligte Einrichtung: Schulich School of Medicine & Dentistry, Western University, London, Ontario, Canada. Hauptthemen laut MEDLINE: Critical Care, Comprehension. Die Arbeit wurde bisher 26-mal zitiert (Crossref) und nennt 23 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Background
We assessed 2 versions of the large language model (LLM) ChatGPT—versions 3.5 and 4.0—in generating appropriate, consistent, and readable recommendations on core critical care topics. Research Question: How do successive large language models compare in terms of generating appropriate, consistent, and readable recommendations on core critical care topics? Design and Methods: A set of 50 LLM-generated responses to clinical questions were evaluated by 2 independent intensivists based on a 5-point Likert scale for appropriateness, consistency, and readability.
Results
ChatGPT 4.0 showed significantly higher median appropriateness scores compared to ChatGPT 3.5 (4.0 vs 3.0, P < .001). However, there was no significant difference in consistency between the 2 versions (40% vs 28%, P = 0.291). Readability, assessed by the Flesch-Kincaid Grade Level, was also not significantly different between the 2 models (14.3 vs 14.4, P = 0.93).
Interpretation
Both models produced “hallucinations”—misinformation delivered with high confidence—which highlights the risk of relying on these tools without domain expertise. Despite potential for clinical application, both models lacked consistency producing different results when asked the same question multiple times. The study underscores the need for clinicians to understand the strengths and limitations of LLMs for safe and effective implementation in critical care settings. Registration: https://osf.io/8chj7/
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Schulich School of Medicine & Dentistry, Western University, London, Ontario, Canada.
- Division of Vascular Surgery, Department of Surgery, University of Toronto, Toronto, Ontario, Canada.
- Division of Critical Care, London Health Sciences Centre, Western University, London, Ontario, Canada.
- Department of Surgery, Trauma Program, London Health Sciences Centre, London, Ontario, Canada.
ORCID-Profile: Kaan Y Balta, Robert Arntfield
Zitieren
APA
Balta, K. Y., Javidan, A. P., Walser, E., Arntfield, R., & Prager, R. (2025). Evaluating the Appropriateness, Consistency, and Readability of ChatGPT in Critical Care Recommendations. Journal of Intensive Care Medicine, 40(2), 184-190. https://doi.org/10.1177/08850666241267871
Vancouver
Balta KY, Javidan AP, Walser E, Arntfield R, Prager R. Evaluating the Appropriateness, Consistency, and Readability of ChatGPT in Critical Care Recommendations. Journal of Intensive Care Medicine. 2025;40(2):184-190. doi:10.1177/08850666241267871
BibTeX
@article{Balta2025,
author = {Balta, Kaan Y and Javidan, Arshia P and Walser, Eric and Arntfield, Robert and Prager, Ross},
title = {Evaluating the Appropriateness, Consistency, and Readability of ChatGPT in Critical Care Recommendations},
journal = {Journal of Intensive Care Medicine},
year = {2025},
volume = {40},
number = {2},
pages = {184--190},
doi = {10.1177/08850666241267871}
}
EUVIMED