EUVIMEDEUVIMED entdecken
Open Access

Evaluation Framework of Large Language Models in Medical Documentation: Development and Usability Study

Junhyuk Seo iD, Dasol Choi iD, Taerim Kim iD, Won Chul Cha iD, Minha Kim iD, Haanju Yoo iD, Namkee Oh iD, YongJin Yi iD, Kye Hwa Lee iD, Edward Choi iD

Journal of Medical Internet Research · 20.11.2024 · 26 · e58329

DOI: 10.2196/58329 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 20.11.2024. Verfasst von Junhyuk Seo und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Digital Health, Samsung Advanced Institute of Health Sciences and Technology (SAIHST), Sungkyunkwan University, Seoul, Republic of Kore…. Hauptthemen laut MEDLINE: Documentation. Die Arbeit wurde bisher 61-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The advancement of large language models (LLMs) offers significant opportunities for health care, particularly in the generation of medical documentation. However, challenges related to ensuring the accuracy and reliability of LLM outputs, coupled with the absence of established quality standards, have raised concerns about their clinical application.

Objective

This study aimed to develop and validate an evaluation framework for assessing the accuracy and clinical applicability of LLM-generated emergency department (ED) records, aiming to enhance artificial intelligence integration in health care documentation.

Methods

We organized the Healthcare Prompt-a-thon, a competitive event designed to explore the capabilities of LLMs in generating accurate medical records. The event involved 52 participants who generated 33 initial ED records using HyperCLOVA X, a Korean-specialized LLM. We applied a dual evaluation approach. First, clinical evaluation: 4 medical professionals evaluated the records using a 5-point Likert scale across 5 criteria—appropriateness, accuracy, structure/format, conciseness, and clinical validity. Second, quantitative evaluation: We developed a framework to categorize and count errors in the LLM outputs, identifying 7 key error types. Statistical methods, including Pearson correlation and intraclass correlation coefficients (ICC), were used to assess consistency and agreement among evaluators.

Results

The clinical evaluation demonstrated strong interrater reliability, with ICC values ranging from 0.653 to 0.887 (P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

DocumentationHumansRepublic of KoreaEmergency Service, HospitalElectronic Health RecordsReproducibility of ResultsArtificial Intelligence

Stichwörter der Autor:innen

artificial intelligenceclinical evaluationemergency departmenthealth care documentationlarge language modelsmedical record accuracy

Autor:innen und Institutionen

  • Department of Digital Health, Samsung Advanced Institute of Health Sciences and Technology (SAIHST), Sungkyunkwan University, Seoul, Republic of Korea.
  • Department of Nursing, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea.
  • Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea.
  • NAVER Digital Healthcare Lab, Seongnam, Republic of Korea.
  • Department of Surgery, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea.
  • Department of Internal Medicine, College of Medicine, Dankook University, Cheonan, Republic of Korea.

ORCID-Profile: Junhyuk Seo, Dasol Choi, Taerim Kim, Won Chul Cha, Minha Kim, Haanju Yoo, Namkee Oh, YongJin Yi, Kye Hwa Lee, Edward Choi

Zitieren

APA

Seo, J., Choi, D., Kim, T., Cha, W. C., Kim, M., Yoo, H., Oh, N., Yi, Y., Lee, K. H., & Choi, E. (2024). Evaluation Framework of Large Language Models in Medical Documentation: Development and Usability Study. Journal of Medical Internet Research, 26, e58329. https://doi.org/10.2196/58329

Vancouver

Seo J, Choi D, Kim T, Cha WC, Kim M, Yoo H, et al. Evaluation Framework of Large Language Models in Medical Documentation: Development and Usability Study. Journal of Medical Internet Research. 2024;26:e58329. doi:10.2196/58329

BibTeX
@article{Seo2024,
  author = {Seo, Junhyuk and Choi, Dasol and Kim, Taerim and Cha, Won Chul and Kim, Minha and Yoo, Haanju and Oh, Namkee and Yi, YongJin and Lee, Kye Hwa and Choi, Edward},
  title = {Evaluation Framework of Large Language Models in Medical Documentation: Development and Usability Study},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e58329},
  doi = {10.2196/58329}
}
Kostenlos starten →