Large Language Models for Chatbot Health Advice Studies
JAMA Network Open · 04.02.2025 · 8 (2) · e2457879
DOI: 10.1001/jamanetworkopen.2024.57879 ↗Kurz gefasst
Systematische Übersicht in der Zeitschrift JAMA Network Open, online erschienen am 04.02.2025. Verfasst von Bright Huo und 12 weiteren Autor:innen. Beteiligte Einrichtung: Division of General Surgery, Department of Surgery, McMaster University, Hamilton, Ontario, Canada. Hauptthemen laut MEDLINE: Artificial Intelligence. Die Arbeit wurde bisher 311-mal zitiert (Crossref) und nennt 193 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Importance There is much interest in the clinical integration of large language models (LLMs) in health care. Many studies have assessed the ability of LLMs to provide health advice, but the quality of their reporting is uncertain.
Objective
To perform a systematic review to examine the reporting variability among peer-reviewed studies evaluating the performance of generative artificial intelligence (AI)–driven chatbots for summarizing evidence and providing health advice to inform the development of the Chatbot Assessment Reporting Tool (CHART). Evidence Review A search of MEDLINE via Ovid, Embase via Elsevier, and Web of Science from inception to October 27, 2023, was conducted with the help of a health sciences librarian to yield 7752 articles. Two reviewers screened articles by title and abstract followed by full-text review to identify primary studies evaluating the clinical accuracy of generative AI-driven chatbots in providing health advice (chatbot health advice studies). Two reviewers then performed data extraction for 137 eligible studies.
Findings
A total of 137 studies were included. Studies examined topics in surgery (55 [40.1%]), medicine (51 [37.2%]), and primary care (13 [9.5%]). Many studies focused on treatment (91 [66.4%]), diagnosis (60 [43.8%]), or disease prevention (29 [21.2%]). Most studies (136 [99.3%]) evaluated inaccessible, closed-source LLMs and did not provide enough information to identify the version of the LLM under evaluation. All studies lacked a sufficient description of LLM characteristics, including temperature, token length, fine-tuning availability, layers, and other details. Most studies (136 [99.3%]) did not describe a prompt engineering phase in their study. The date of LLM querying was reported in 54 (39.4%) studies. Most studies (89 [65.0%]) used subjective means to define the successful performance of the chatbot, while less than one-third addressed the ethical, regulatory, and patient safety implications of the clinical integration of LLMs. Conclusions and Relevance In this systematic review of 137 chatbot health advice studies, the reporting quality was heterogeneous and may inform the development of the CHART reporting standards. Ethical, regulatory, and patient safety considerations are crucial as interest grows in the clinical integration of LLMs.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- Division of General Surgery, Department of Surgery, McMaster University, Hamilton, Ontario, Canada.
- Michael G. DeGroote School of Medicine, McMaster University, Hamilton, Ontario, Canada.
- H. Ross University School of Medicine, Miramar, Florida.
- Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada.
- Institute of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada.
- Hospital Clinico San Carlos, IdISSC, Universidad Complutense de Madrid, Madrid, Spain.
Zitieren
APA
Huo, B., Boyle, A., Marfo, N., Tangamornsuksan, W., Steen, J. P., McKechnie, T., Lee, Y., Mayol, J., Antoniou, S. A., Thirunavukarasu, A. J., Sanger, S., Ramji, K., & Guyatt, G. (2025). Large Language Models for Chatbot Health Advice Studies. JAMA Network Open, 8(2), e2457879. https://doi.org/10.1001/jamanetworkopen.2024.57879
Vancouver
Huo B, Boyle A, Marfo N, Tangamornsuksan W, Steen JP, McKechnie T, et al. Large Language Models for Chatbot Health Advice Studies. JAMA Network Open. 2025;8(2):e2457879. doi:10.1001/jamanetworkopen.2024.57879
BibTeX
@article{Huo2025,
author = {Huo, Bright and Boyle, Amy and Marfo, Nana and Tangamornsuksan, Wimonchat and Steen, Jeremy P and McKechnie, Tyler and Lee, Yung and Mayol, Julio and Antoniou, Stavros A and Thirunavukarasu, Arun James and Sanger, Stephanie and Ramji, Karim and Guyatt, Gordon},
title = {Large Language Models for Chatbot Health Advice Studies},
journal = {JAMA Network Open},
year = {2025},
volume = {8},
number = {2},
pages = {e2457879},
doi = {10.1001/jamanetworkopen.2024.57879}
}
EUVIMED