EUVIMEDEUVIMED entdecken
Crossref

Development and Evaluation of an Instruction-Tuned Large Language Model for Reply Letter Generation from Multiple Clinical Document Types

Eunbeen Jo, Yebin Lee, Eunbi Cho, Sujin Kim, Eunyeong Park, Hyeonseo Ryu, Hyung Joon Joo iD, Sanghoun Song

Journal of Medical Systems · 29.10.2026 · 50 (1)

DOI: 10.1007/s10916-026-02483-1 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Systems, online erschienen am 29.10.2026. Verfasst von Eunbeen Jo und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Korea University College of Medicine, Seoul, Republic of Korea. Hauptthemen laut MEDLINE: Large Language Models, Electronic Health Records, Correspondence as Topic. Die Arbeit nennt 58 Literaturangaben. Gefördert von: MSIT(Ministry of Science and ICT), Korea.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Medical documentation, including reply letters from specialty to primary care, imposes a substantial burden on clinicians. Recent advances in large language models (LLMs) may help alleviate this burden. This study aims to develop an instruction-tuned LLM to generate reply letters by integrating five types of electronic medical record documents: admission notes, progress notes, procedure notes, operative notes, and discharge summaries. A total of 8,527 hospitalization episodes from the cardiology and thoracic surgery departments of a tertiary hospital were used for training, and 105 episodes were allocated to the test set. Mistral-Nemo-Instruct-2407 was fine-tuned using the instruction-tuning dataset generated with GPT-4.1. Model performance was evaluated using a dual framework combining expert assessment and a GPT-based evaluation. Experts rated human-written letters higher in accuracy (4.77 vs. 4.21, p < 0.001) and appropriateness (4.90 vs. 4.63, p < 0.001), whereas GPT-based evaluations rated AI-generated letters higher across all dimensions. A hybrid scoring approach integrating expert and GPT-based evaluations improved score consistency among evaluators; in an exploratory comparison using a common weighting, the largest proportion of cases was classified as comparable between the two letter types across all quality dimensions. Expert ratings for workflow suitability (M = 4.30) indicated positive expectations for practical implementation. These findings suggest that LLM-based reply letter generation may serve as a supportive tool for drafting clinical correspondence, although its safety and impact on documentation burden require prospective evaluation in real-world clinical workflows.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsElectronic Health RecordsCorrespondence as TopicHumansGenerative Artificial Intelligence

Stichwörter der Autor:innen

Clinical communicationEMR dataInstruction-tuned language modelsReply letters

Autor:innen und Institutionen

  • Department of Biomedical Informatics, Korea University College of Medicine, Seoul, Republic of Korea.
  • Korea University Research Institute for Medical Bigdata Science, Korea University College of Medicine, Seoul, Republic of Korea.
  • Department of Linguistics, Korea University, Seoul, Republic of Korea.
  • Department of Cardiology, Korea University Anam Hospital, Seoul, Republic of Korea.
  • Department of Biomedical Informatics, Korea University College of Medicine, Seoul, Republic of Korea. drjoohj@korea.ac.kr.
  • Department of Cardiology, Korea University Anam Hospital, Seoul, Republic of Korea. drjoohj@korea.ac.kr.

ORCID-Profile: Hyung Joon Joo

Förderung

  • MSIT(Ministry of Science and ICT), Korea

Zitieren

APA

Jo, E., Lee, Y., Cho, E., Kim, S., Park, E., Ryu, H., Joo, H. J., & Song, S. (2026). Development and Evaluation of an Instruction-Tuned Large Language Model for Reply Letter Generation from Multiple Clinical Document Types. Journal of Medical Systems, 50(1). https://doi.org/10.1007/s10916-026-02483-1

Vancouver

Jo E, Lee Y, Cho E, Kim S, Park E, Ryu H, et al. Development and Evaluation of an Instruction-Tuned Large Language Model for Reply Letter Generation from Multiple Clinical Document Types. Journal of Medical Systems. 2026;50(1). doi:10.1007/s10916-026-02483-1

BibTeX
@article{Jo2026,
  author = {Jo, Eunbeen and Lee, Yebin and Cho, Eunbi and Kim, Sujin and Park, Eunyeong and Ryu, Hyeonseo and Joo, Hyung Joon and Song, Sanghoun},
  title = {Development and Evaluation of an Instruction-Tuned Large Language Model for Reply Letter Generation from Multiple Clinical Document Types},
  journal = {Journal of Medical Systems},
  year = {2026},
  volume = {50},
  number = {1},
  doi = {10.1007/s10916-026-02483-1}
}
Kostenlos starten →