Evaluation Framework of Large Language Models in Medical Documentation: Development and Usability Study
Journal of Medical Internet Research · 20.11.2024 · 26 · e58329
DOI: 10.2196/58329 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 20.11.2024. Verfasst von Junhyuk Seo und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Digital Health, Samsung Advanced Institute of Health Sciences and Technology (SAIHST), Sungkyunkwan University, Seoul, Republic of Kore…. Hauptthemen laut MEDLINE: Documentation. Die Arbeit wurde bisher 61-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
The advancement of large language models (LLMs) offers significant opportunities for health care, particularly in the generation of medical documentation. However, challenges related to ensuring the accuracy and reliability of LLM outputs, coupled with the absence of established quality standards, have raised concerns about their clinical application.
Objective
This study aimed to develop and validate an evaluation framework for assessing the accuracy and clinical applicability of LLM-generated emergency department (ED) records, aiming to enhance artificial intelligence integration in health care documentation.
Methods
We organized the Healthcare Prompt-a-thon, a competitive event designed to explore the capabilities of LLMs in generating accurate medical records. The event involved 52 participants who generated 33 initial ED records using HyperCLOVA X, a Korean-specialized LLM. We applied a dual evaluation approach. First, clinical evaluation: 4 medical professionals evaluated the records using a 5-point Likert scale across 5 criteria—appropriateness, accuracy, structure/format, conciseness, and clinical validity. Second, quantitative evaluation: We developed a framework to categorize and count errors in the LLM outputs, identifying 7 key error types. Statistical methods, including Pearson correlation and intraclass correlation coefficients (ICC), were used to assess consistency and agreement among evaluators.
Results
The clinical evaluation demonstrated strong interrater reliability, with ICC values ranging from 0.653 to 0.887 (P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Digital Health, Samsung Advanced Institute of Health Sciences and Technology (SAIHST), Sungkyunkwan University, Seoul, Republic of Korea.
- Department of Nursing, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea.
- Department of Emergency Medicine, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea.
- NAVER Digital Healthcare Lab, Seongnam, Republic of Korea.
- Department of Surgery, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea.
- Department of Internal Medicine, College of Medicine, Dankook University, Cheonan, Republic of Korea.
ORCID-Profile: Junhyuk Seo, Dasol Choi, Taerim Kim, Won Chul Cha, Minha Kim, Haanju Yoo, Namkee Oh, YongJin Yi, Kye Hwa Lee, Edward Choi
Zitieren
APA
Seo, J., Choi, D., Kim, T., Cha, W. C., Kim, M., Yoo, H., Oh, N., Yi, Y., Lee, K. H., & Choi, E. (2024). Evaluation Framework of Large Language Models in Medical Documentation: Development and Usability Study. Journal of Medical Internet Research, 26, e58329. https://doi.org/10.2196/58329
Vancouver
Seo J, Choi D, Kim T, Cha WC, Kim M, Yoo H, et al. Evaluation Framework of Large Language Models in Medical Documentation: Development and Usability Study. Journal of Medical Internet Research. 2024;26:e58329. doi:10.2196/58329
BibTeX
@article{Seo2024,
author = {Seo, Junhyuk and Choi, Dasol and Kim, Taerim and Cha, Won Chul and Kim, Minha and Yoo, Haanju and Oh, Namkee and Yi, YongJin and Lee, Kye Hwa and Choi, Edward},
title = {Evaluation Framework of Large Language Models in Medical Documentation: Development and Usability Study},
journal = {Journal of Medical Internet Research},
year = {2024},
volume = {26},
pages = {e58329},
doi = {10.2196/58329}
}
EUVIMED