Development and Evaluation of an Instruction-Tuned Large Language Model for Reply Letter Generation from Multiple Clinical Document Types
Journal of Medical Systems · 29.10.2026 · 50 (1)
DOI: 10.1007/s10916-026-02483-1 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Medical Systems, online erschienen am 29.10.2026. Verfasst von Eunbeen Jo und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Korea University College of Medicine, Seoul, Republic of Korea. Hauptthemen laut MEDLINE: Large Language Models, Electronic Health Records, Correspondence as Topic. Die Arbeit nennt 58 Literaturangaben. Gefördert von: MSIT(Ministry of Science and ICT), Korea.
Abstract (Originalsprache)
Medical documentation, including reply letters from specialty to primary care, imposes a substantial burden on clinicians. Recent advances in large language models (LLMs) may help alleviate this burden. This study aims to develop an instruction-tuned LLM to generate reply letters by integrating five types of electronic medical record documents: admission notes, progress notes, procedure notes, operative notes, and discharge summaries. A total of 8,527 hospitalization episodes from the cardiology and thoracic surgery departments of a tertiary hospital were used for training, and 105 episodes were allocated to the test set. Mistral-Nemo-Instruct-2407 was fine-tuned using the instruction-tuning dataset generated with GPT-4.1. Model performance was evaluated using a dual framework combining expert assessment and a GPT-based evaluation. Experts rated human-written letters higher in accuracy (4.77 vs. 4.21, p < 0.001) and appropriateness (4.90 vs. 4.63, p < 0.001), whereas GPT-based evaluations rated AI-generated letters higher across all dimensions. A hybrid scoring approach integrating expert and GPT-based evaluations improved score consistency among evaluators; in an exploratory comparison using a common weighting, the largest proportion of cases was classified as comparable between the two letter types across all quality dimensions. Expert ratings for workflow suitability (M = 4.30) indicated positive expectations for practical implementation. These findings suggest that LLM-based reply letter generation may serve as a supportive tool for drafting clinical correspondence, although its safety and impact on documentation burden require prospective evaluation in real-world clinical workflows.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Informatics, Korea University College of Medicine, Seoul, Republic of Korea.
- Korea University Research Institute for Medical Bigdata Science, Korea University College of Medicine, Seoul, Republic of Korea.
- Department of Linguistics, Korea University, Seoul, Republic of Korea.
- Department of Cardiology, Korea University Anam Hospital, Seoul, Republic of Korea.
- Department of Biomedical Informatics, Korea University College of Medicine, Seoul, Republic of Korea. drjoohj@korea.ac.kr.
- Department of Cardiology, Korea University Anam Hospital, Seoul, Republic of Korea. drjoohj@korea.ac.kr.
ORCID-Profile: Hyung Joon Joo
Förderung
- MSIT(Ministry of Science and ICT), Korea
Zitieren
APA
Jo, E., Lee, Y., Cho, E., Kim, S., Park, E., Ryu, H., Joo, H. J., & Song, S. (2026). Development and Evaluation of an Instruction-Tuned Large Language Model for Reply Letter Generation from Multiple Clinical Document Types. Journal of Medical Systems, 50(1). https://doi.org/10.1007/s10916-026-02483-1
Vancouver
Jo E, Lee Y, Cho E, Kim S, Park E, Ryu H, et al. Development and Evaluation of an Instruction-Tuned Large Language Model for Reply Letter Generation from Multiple Clinical Document Types. Journal of Medical Systems. 2026;50(1). doi:10.1007/s10916-026-02483-1
BibTeX
@article{Jo2026,
author = {Jo, Eunbeen and Lee, Yebin and Cho, Eunbi and Kim, Sujin and Park, Eunyeong and Ryu, Hyeonseo and Joo, Hyung Joon and Song, Sanghoun},
title = {Development and Evaluation of an Instruction-Tuned Large Language Model for Reply Letter Generation from Multiple Clinical Document Types},
journal = {Journal of Medical Systems},
year = {2026},
volume = {50},
number = {1},
doi = {10.1007/s10916-026-02483-1}
}
EUVIMED