Systematic review: The use of large language models as medical chatbots in digestive diseases
Alimentary Pharmacology & Therapeutics · 2024-07 · 60 (2) · 144-166
DOI: 10.1111/apt.18058 ↗Kurz gefasst
Systematische Übersicht in der Zeitschrift Alimentary Pharmacology & Therapeutics, online erschienen am 27.05.2024. Verfasst von Mauro Giuffrè und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Internal Medicine (Digestive Diseases), Yale School of Medicine, New Haven, Connecticut, USA. Hauptthemen laut MEDLINE: Gastroenterology. Die Arbeit wurde bisher 59-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIDDK NIH HHS.
Abstract (Originalsprache)
Background
Interest in large language models (LLMs), such as OpenAI's ChatGPT, across multiple specialties has grown as a source of patient‐facing medical advice and provider‐facing clinical decision support. The accuracy of LLM responses for gastroenterology and hepatology‐related questions is unknown.
Aims
To evaluate the accuracy and potential safety implications for LLMs for the diagnosis, management and treatment of questions related to gastroenterology and hepatology.
Methods
We conducted a systematic literature search including Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus and the Web of Science Core Collection to identify relevant articles published from inception until January 28, 2024, using a combination of keywords and controlled vocabulary for LLMs and gastroenterology or hepatology. Accuracy was defined as the percentage of entirely correct answers.
Results
Among the 1671 reports screened, we identified 33 full‐text articles on using LLMs in gastroenterology and hepatology and included 18 in the final analysis. The accuracy of question‐responding varied across different model versions. For example, accuracy ranged from 6.4% to 45.5% with ChatGPT‐3.5 and was between 40% and 91.4% with ChatGPT‐4. In addition, the absence of standardised methodology and reporting metrics for studies involving LLMs places all the studies at a high risk of bias and does not allow for the generalisation of single‐study results.
Conclusions
Current general‐purpose LLMs have unacceptably low accuracy on clinical gastroenterology and hepatology tasks, which may lead to adverse patient safety events through incorrect information or triage recommendations, which might overburden healthcare systems or delay necessary care.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Department of Internal Medicine (Digestive Diseases), Yale School of Medicine, New Haven, Connecticut, USA.
- Department of Medical, Surgical and Health Sciences, University of Trieste, Trieste, Italy.
- Department of Engineering and Architecture, University of Trieste, Trieste, Italy.
- Department of Mathematics at Baruch College, City University of new York, New York, New York, USA.
- Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, USA.
- Research & Education Librarian (Clinical) at Harvey Cushing/John Hay Whitney Medical Library, Yale University, New Haven, Connecticut, USA.
ORCID-Profile: Mauro Giuffrè, Dennis Legen Shung
Förderung
- NIDDK NIH HHS
Zitieren
APA
Giuffrè, M., Kresevic, S., You, K., Dupont, J., Huebner, J., Grimshaw, A. A., & Shung, D. L. (2024). Systematic review: The use of large language models as medical chatbots in digestive diseases. Alimentary Pharmacology & Therapeutics, 60(2), 144-166. https://doi.org/10.1111/apt.18058
Vancouver
Giuffrè M, Kresevic S, You K, Dupont J, Huebner J, Grimshaw AA, et al. Systematic review: The use of large language models as medical chatbots in digestive diseases. Alimentary Pharmacology & Therapeutics. 2024;60(2):144-166. doi:10.1111/apt.18058
BibTeX
@article{Giuffr2024,
author = {Giuffrè, Mauro and Kresevic, Simone and You, Kisung and Dupont, Johannes and Huebner, Jack and Grimshaw, Alyssa Ann and Shung, Dennis Legen},
title = {Systematic review: The use of large language models as medical chatbots in digestive diseases},
journal = {Alimentary Pharmacology & Therapeutics},
year = {2024},
volume = {60},
number = {2},
pages = {144--166},
doi = {10.1111/apt.18058}
}
EUVIMED