Performance of ChatGPT-4o in the diagnostic workup of fever among returning travellers requiring hospitalization: a validation study
Journal of Travel Medicine · 25.04.2025 · 32 (4)
DOI: 10.1093/jtm/taaf005 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift Journal of Travel Medicine, online erschienen am 07.02.2025. Verfasst von Dana Yelin und 7 weiteren Autor:innen. Beteiligte Einrichtung: Infectious Diseases Unit, Sheba Medical Center, Tel HaShomer, Ramat Gan, Israel. Hauptthemen laut MEDLINE: Fever, Travel. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 28 Literaturangaben.
Abstract (Originalsprache)
Background
Febrile illness in returned travellers presents a diagnostic challenge in non-endemic settings. Chat generative pretrained transformer (ChatGPT) has the potential to assist in medical tasks, yet its diagnostic performance in clinical settings has rarely been evaluated. We conducted a validation assessment of ChatGPT-4o’s performance in the workup of fever in returning travellers.
Methods
We retrieved the medical records of returning travellers hospitalized with fever during 2009–2024. Their clinical scenarios at time of presentation to the emergency department were prompted to ChatGPT-4o, using a detailed uniform format. The model was further prompted with four consistent questions concerning the differential diagnosis and recommended workup. To avoid training, we kept the model blinded to the final diagnosis. Our primary outcome was ChatGPT-4o’s success rates in predicting the final diagnosis when requested to specify the top three differential diagnoses. Secondary outcomes were success rates when prompted to specify the single most likely diagnosis, and all necessary diagnostics. We also assessed ChatGPT-4o as a predicting tool for malaria and qualitatively evaluated its failures.
Results
ChatGPT-4o predicted the final diagnosis in 68% [95% confidence interval (CI) 59–77%], 78% (95% CI 69–85%) and 83% (95% CI 74–89%) of the 114 cases, when prompted to specify the most likely diagnosis, top three diagnoses and all possible diagnoses, respectively. ChatGPT-4o showed a sensitivity of 100% (95% CI 93–100%) and a specificity of 94% (95% CI 85–98%) for predicting malaria. The model failed to provide the final diagnosis in 18% (20/114) of cases, primarily by failing to predict globally endemic infections (16/21, 76%).
Conclusions
ChatGPT-4o demonstrated high diagnostic accuracy when prompted with real-life scenarios of febrile returning travellers presenting to the emergency department, especially for malaria. Model training is expected to yield an improved performance and facilitate diagnostic decision-making in the field.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Infectious Diseases Unit, Sheba Medical Center, Tel HaShomer, Ramat Gan, Israel.
- School of Medicine, Faculty of Medical and Health Sciences, Tel Aviv University, P.O.B 39040 Tel Aviv 69978, Israel.
- Internal Medicine C, Sheba Medical Center, Tel HaShomer, Ramat Gan, Israel.
- Infection Control and Prevention Unit, Sheba Medical Center, Tel HaShomer, Ramat Gan, Israel.
- Center for Geographic Medicine, Sheba Medical Center, Tel HaShomer, Ramat Gan, Israel.
ORCID-Profile: Ili Margalit
Zitieren
APA
Yelin, D., Shirin, N., Harris, I., Peretz, Y., Yahav, D., Schwartz, E., Leshem, E., & Margalit, I. (2025). Performance of ChatGPT-4o in the diagnostic workup of fever among returning travellers requiring hospitalization: a validation study. Journal of Travel Medicine, 32(4). https://doi.org/10.1093/jtm/taaf005
Vancouver
Yelin D, Shirin N, Harris I, Peretz Y, Yahav D, Schwartz E, et al. Performance of ChatGPT-4o in the diagnostic workup of fever among returning travellers requiring hospitalization: a validation study. Journal of Travel Medicine. 2025;32(4). doi:10.1093/jtm/taaf005
BibTeX
@article{Yelin2025,
author = {Yelin, Dana and Shirin, Neta and Harris, Itai and Peretz, Yovel and Yahav, Dafna and Schwartz, Eli and Leshem, Eyal and Margalit, Ili},
title = {Performance of ChatGPT-4o in the diagnostic workup of fever among returning travellers requiring hospitalization: a validation study},
journal = {Journal of Travel Medicine},
year = {2025},
volume = {32},
number = {4},
doi = {10.1093/jtm/taaf005}
}
EUVIMED