EUVIMEDEUVIMED entdecken
Systematische ÜbersichtOpen Access · CC BY 4.0

Systematic review: The use of large language models as medical chatbots in digestive diseases

Mauro Giuffrè iD, Simone Kresevic, Kisung You, Johannes Dupont, Jack Huebner, Alyssa Ann Grimshaw, Dennis Legen Shung iD

Alimentary Pharmacology & Therapeutics · 2024-07 · 60 (2) · 144-166

DOI: 10.1111/apt.18058 ↗

Kurz gefasst

Systematische Übersicht in der Zeitschrift Alimentary Pharmacology & Therapeutics, online erschienen am 27.05.2024. Verfasst von Mauro Giuffrè und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Internal Medicine (Digestive Diseases), Yale School of Medicine, New Haven, Connecticut, USA. Hauptthemen laut MEDLINE: Gastroenterology. Die Arbeit wurde bisher 59-mal zitiert (Crossref) und nennt 62 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIDDK NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Interest in large language models (LLMs), such as OpenAI's ChatGPT, across multiple specialties has grown as a source of patient‐facing medical advice and provider‐facing clinical decision support. The accuracy of LLM responses for gastroenterology and hepatology‐related questions is unknown.

Aims

To evaluate the accuracy and potential safety implications for LLMs for the diagnosis, management and treatment of questions related to gastroenterology and hepatology.

Methods

We conducted a systematic literature search including Cochrane Library, Google Scholar, Ovid Embase, Ovid MEDLINE, PubMed, Scopus and the Web of Science Core Collection to identify relevant articles published from inception until January 28, 2024, using a combination of keywords and controlled vocabulary for LLMs and gastroenterology or hepatology. Accuracy was defined as the percentage of entirely correct answers.

Results

Among the 1671 reports screened, we identified 33 full‐text articles on using LLMs in gastroenterology and hepatology and included 18 in the final analysis. The accuracy of question‐responding varied across different model versions. For example, accuracy ranged from 6.4% to 45.5% with ChatGPT‐3.5 and was between 40% and 91.4% with ChatGPT‐4. In addition, the absence of standardised methodology and reporting metrics for studies involving LLMs places all the studies at a high risk of bias and does not allow for the generalisation of single‐study results.

Conclusions

Current general‐purpose LLMs have unacceptably low accuracy on clinical gastroenterology and hepatology tasks, which may lead to adverse patient safety events through incorrect information or triage recommendations, which might overburden healthcare systems or delay necessary care.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

GastroenterologyHumansDigestive System DiseasesDecision Support Systems, ClinicalLanguage

Autor:innen und Institutionen

  • Department of Internal Medicine (Digestive Diseases), Yale School of Medicine, New Haven, Connecticut, USA.
  • Department of Medical, Surgical and Health Sciences, University of Trieste, Trieste, Italy.
  • Department of Engineering and Architecture, University of Trieste, Trieste, Italy.
  • Department of Mathematics at Baruch College, City University of new York, New York, New York, USA.
  • Department of Internal Medicine, Yale School of Medicine, New Haven, Connecticut, USA.
  • Research & Education Librarian (Clinical) at Harvey Cushing/John Hay Whitney Medical Library, Yale University, New Haven, Connecticut, USA.

ORCID-Profile: Mauro Giuffrè, Dennis Legen Shung

Förderung

  • NIDDK NIH HHS

Zitieren

APA

Giuffrè, M., Kresevic, S., You, K., Dupont, J., Huebner, J., Grimshaw, A. A., & Shung, D. L. (2024). Systematic review: The use of large language models as medical chatbots in digestive diseases. Alimentary Pharmacology & Therapeutics, 60(2), 144-166. https://doi.org/10.1111/apt.18058

Vancouver

Giuffrè M, Kresevic S, You K, Dupont J, Huebner J, Grimshaw AA, et al. Systematic review: The use of large language models as medical chatbots in digestive diseases. Alimentary Pharmacology & Therapeutics. 2024;60(2):144-166. doi:10.1111/apt.18058

BibTeX
@article{Giuffr2024,
  author = {Giuffrè, Mauro and Kresevic, Simone and You, Kisung and Dupont, Johannes and Huebner, Jack and Grimshaw, Alyssa Ann and Shung, Dennis Legen},
  title = {Systematic review: The use of large language models as medical chatbots in digestive diseases},
  journal = {Alimentary Pharmacology & Therapeutics},
  year = {2024},
  volume = {60},
  number = {2},
  pages = {144--166},
  doi = {10.1111/apt.18058}
}
Kostenlos starten →