EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating large language models for drafting emergency department encounter summaries

Christopher Y K Williams iD, Jaskaran Bains, Tianyu Tang, Kishan Patel, Alexa N Lucas, Fiona Chen, Brenda Y Miao, Atul J Butte, Aaron E Kornblith iD

PLOS Digital Health · 17.06.2025 · 4 (6) · e0000899

DOI: 10.1371/journal.pdig.0000899 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS Digital Health, online erschienen am 17.06.2025. Verfasst von Christopher Y K Williams und 8 weiteren Autor:innen. Beteiligte Einrichtung: Bakar Computational Health Sciences Institute, University of California, San Francisco, California, United States of America. Die Arbeit wurde bisher 51-mal zitiert (Crossref) und nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NICHD NIH HHS.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) possess a range of capabilities which may be applied to the clinical domain, including text summarization. As ambient artificial intelligence scribes and other LLM-based tools begin to be deployed within healthcare settings, rigorous evaluations of the accuracy of these technologies are urgently needed. In this cross-sectional study of 100 randomly sampled adult Emergency Department (ED) visits from 2012 to 2023 at the University of California, San Francisco ED, we sought to investigate the performance of GPT-4 and GPT-3.5-turbo in generating ED encounter summaries and evaluate the prevalence and type of errors for each section of the encounter summary across three evaluation criteria: 1) Inaccuracy of LLM-summarized information; 2) Hallucination of information; 3) Omission of relevant clinical information. In total, 33% of summaries generated by GPT-4 and 10% of those generated by GPT-3.5-turbo were entirely error-free across all evaluated domains. Summaries generated by GPT-4 were mostly accurate, with inaccuracies found in only 10% of cases, however, 42% of the summaries exhibited hallucinations and 47% omitted clinically relevant information. Inaccuracies and hallucinations were most commonly found in the Plan sections of LLM-generated summaries, while clinical omissions were concentrated in text describing patients’ Physical Examination findings or History of Presenting Complaint. The potential harmfulness score across errors was low, with a mean score of 0.57 (SD 1.11) out of 7 and only three errors scoring 4 (‘Potential for permanent harm’) or greater. In summary, we found that LLMs could generate accurate encounter summaries but were liable to hallucination and omission of clinically relevant information. Individual errors on average had a low potential for harm. A comprehensive understanding of the location and type of errors found in LLM-generated clinical text is important to facilitate clinician review of such content and prevent patient harm.

Autor:innen und Institutionen

  • Bakar Computational Health Sciences Institute, University of California, San Francisco, California, United States of America.
  • Department of Emergency Medicine, University of California, San Francisco, California, United States of America.

ORCID-Profile: Christopher Y K Williams, Aaron E Kornblith

Förderung

  • NICHD NIH HHS

Zitieren

APA

Williams, C. Y. K., Bains, J., Tang, T., Patel, K., Lucas, A. N., Chen, F., Miao, B. Y., Butte, A. J., & Kornblith, A. E. (2025). Evaluating large language models for drafting emergency department encounter summaries. PLOS Digital Health, 4(6), e0000899. https://doi.org/10.1371/journal.pdig.0000899

Vancouver

Williams CYK, Bains J, Tang T, Patel K, Lucas AN, Chen F, et al. Evaluating large language models for drafting emergency department encounter summaries. PLOS Digital Health. 2025;4(6):e0000899. doi:10.1371/journal.pdig.0000899

BibTeX
@article{Williams2025,
  author = {Williams, Christopher Y K and Bains, Jaskaran and Tang, Tianyu and Patel, Kishan and Lucas, Alexa N and Chen, Fiona and Miao, Brenda Y and Butte, Atul J and Kornblith, Aaron E},
  title = {Evaluating large language models for drafting emergency department encounter summaries},
  journal = {PLOS Digital Health},
  year = {2025},
  volume = {4},
  number = {6},
  pages = {e0000899},
  doi = {10.1371/journal.pdig.0000899}
}
Kostenlos starten →