Systematic review: The use of large language models as medical chatbots in digestive diseases
Alimentary Pharmacology & Therapeutics · 2024-07 · 60 (2) · 144-166
DOI: 10.1111/apt.18058 ↗At a glance
Systematische Übersicht in Alimentary Pharmacology & Therapeutics, published online on 2024-05-27. Written by Mauro Giuffrè and 6 co-authors. Institution: Department of Internal Medicine (Digestive Diseases), Yale School of Medicine, New Haven, Connecticut, USA. Major MEDLINE topics: Gastroenterology. Cited 59 times so far (Crossref); 62 references. The full text is freely available (open access, CC BY 4.0). Funded by: NIDDK NIH HHS.
Abstract (original language)
Background
Interest in large language models (LLMs), such as OpenAI's ChatGPT, across multiple specialties has grown as a source of patient‐facing medical advice and provider‐facing clinical decision support. The accuracy of LLM responses for gastroenterology and hepatology‐related questions is unknown.
Aims
To evaluate the accuracy and potential safety implications for LLMs for the diagnosis, management and treatment of questions related to gastroenterology and hepatology.
Methods
We conducted a systematic literature search including Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus and the Web of Science Core Collection to identify relevant articles published from inception until January 28, 2024, using a combination of keywords and controlled vocabulary for LLMs and gastroenterology or hepatology. Accuracy was defined as the percentage of entirely correct answers.
Results
Among the 1671 reports screened, we identified 33 full‐text articles on using LLMs in gastroenterology and hepatology and included 18 in the final analysis. The accuracy of question‐responding varied across different model versions. For example, accuracy ranged from 6.4% to 45.5% with ChatGPT‐3.5 and was between 40% and 91.4% with ChatGPT‐4. In addition, the absence of standardised methodology and reporting metrics for studies involving LLMs places all the studies at a high risk of bias and does not allow for the generalisation of single‐study results.
Conclusions
Current general‐purpose LLMs have unacceptably low accuracy on clinical gastroenterology and hepatology tasks, which may lead to adverse patient safety events through incorrect information or triage recommendations, which might overburden healthcare systems or delay necessary care.
Subjects
MeSH terms (MEDLINE)
Authors and institutions
- Department of Internal Medicine (Digestive Diseases), Yale School of Medicine, New Haven, Connecticut, USA.
- Department of Medical, Surgical and Health Sciences, University of Trieste, Trieste, Italy.
- Department of Engineering and Architecture, University of Trieste, Trieste, Italy.
- Department of Mathematics at Baruch College, City University of new York, New York, New York, USA.
- Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, USA.
- Research & Education Librarian (Clinical) at Harvey Cushing/John Hay Whitney Medical Library, Yale University, New Haven, Connecticut, USA.
ORCID profiles: Mauro Giuffrè, Dennis Legen Shung
Funding
- NIDDK NIH HHS
Cite
APA
Giuffrè, M., Kresevic, S., You, K., Dupont, J., Huebner, J., Grimshaw, A. A., & Shung, D. L. (2024). Systematic review: The use of large language models as medical chatbots in digestive diseases. Alimentary Pharmacology & Therapeutics, 60(2), 144-166. https://doi.org/10.1111/apt.18058
Vancouver
Giuffrè M, Kresevic S, You K, Dupont J, Huebner J, Grimshaw AA, et al. Systematic review: The use of large language models as medical chatbots in digestive diseases. Alimentary Pharmacology & Therapeutics. 2024;60(2):144-166. doi:10.1111/apt.18058
BibTeX
@article{Giuffr2024,
author = {Giuffrè, Mauro and Kresevic, Simone and You, Kisung and Dupont, Johannes and Huebner, Jack and Grimshaw, Alyssa Ann and Shung, Dennis Legen},
title = {Systematic review: The use of large language models as medical chatbots in digestive diseases},
journal = {Alimentary Pharmacology & Therapeutics},
year = {2024},
volume = {60},
number = {2},
pages = {144--166},
doi = {10.1111/apt.18058}
}
EUVIMED