Two-Stage Extraction of Clinical Course Information From Psychiatric Discharge Summaries Using Fine-Tuned Large Language Models: Cross-Validation Study
JMIR Formative Research · 06.08.2026 · 10 · e94454-e94454
DOI: 10.2196/94454 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Formative Research, online erschienen am 06.08.2026. Verfasst von Chien-Hung Chen und 8 weiteren Autor:innen. Beteiligte Einrichtung: Graduate Institute of Networking and Multimedia, National Taiwan University, Taipei, Taiwan. Hauptthemen laut MEDLINE: Large Language Models, Patient Discharge Summaries, Patient Discharge, Mental Disorders. Die Arbeit nennt 20 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Clinical course information, including disease onset, episode recurrence, and hospitalization history, is essential for psychiatric care and research. However, these data are embedded in unstructured clinical narratives with substantial linguistic variability, making manual extraction labor-intensive and rule-based extraction difficult to scale. Fine-tuned large language models (LLMs) may flexibly extract such information from privacy-sensitive psychiatric records.
Objective
This study aimed to evaluate the performance of LLMs for automatically extracting temporal and clinical course information from psychiatric discharge summaries.
Methods
We analyzed 500 psychiatric discharge summaries from the Integrated Medical Database of National Taiwan University Hospital. A psychiatrist and a natural language processing researcher manually annotated clinical events and temporal information. Four open-source LLMs (LLaMA, MentaLLaMA, OpenBioLLM, and Mistral) were fine-tuned using low-rank adaptation and evaluated using 10-fold cross-validation. A 2-stage framework was developed in which sentence-level extraction of clinical events and temporal information was followed by chart-level prediction of 4 clinical course features: first-episode onset time, episode count, number of psychiatric hospitalizations, and most recent hospitalization. Performance was assessed using precision, recall, F 1 -score, accuracy, mean absolute error (MAE), and bootstrap significance testing.
Results
A total of 12,947 sentences were extracted from 500 discharge summaries, yielding 7177 clinical event annotations and 4842 temporal annotations. At the sentence level, Mistral achieved the highest F 1 -scores for clinical event extraction, including symptom/episode detection (0.925, 95% CI 0.920‐0.930), hospitalization detection (0.944, 95% CI 0.934‐0.952), and remission/response detection (0.867, 95% CI 0.851‐0.883). Mistral also achieved the highest F 1 -scores for most temporal information categories, including age expressions (0.983, 95% CI 0.973‐0.993), relative time expressions (0.953, 95% CI 0.944‐0.963), duration expressions (0.976, 95% CI 0.962‐0.987), and vague temporal expressions (0.888, 95% CI 0.871‐0.905). At the chart level, the proposed 2-stage framework showed the clearest benefit for first-episode onset prediction. Using the 2-stage framework, Mistral achieved the highest point estimate for onset accuracy (0.772), although its performance did not significantly differ from that of LLaMA using the same framework (0.744; bootstrap P =.26). For onset prediction, the 2-stage framework significantly outperformed the direct and joint extraction approaches across all evaluated models (all bootstrap P ≤.002). For other chart-level features, the best-performing approach varied: using the 2-stage framework, Mistral achieved the highest episode count accuracy (0.624; MAE=0.428); using the direct approach, LLaMA achieved the highest hospitalization count accuracy (0.692; MAE=0.379); and using the 2-stage framework, OpenBioLLM achieved the highest most recent hospitalization accuracy (0.868; F 1 -score=0.617).
Conclusions
Fine-tuned, locally deployable, open-source LLMs can extract temporal and longitudinal disease course information from psychiatric discharge summaries. The 2-stage framework was most beneficial for first-episode onset prediction and performed competitively across other chart-level features, supporting the use of LLMs to transform heterogeneous psychiatric narratives into structured data for research and decision support.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Graduate Institute of Networking and Multimedia, National Taiwan University, Taipei, Taiwan.
- Institute of Psychiatry, Psychology & Neuroscience, King's College London, London, United Kingdom.
- Department of Electrical Engineering, College of Electrical Engineering and Computer Science, National Kaohsiung University of Science and Technology, Kaohsiung, Taiwan.
- National Center for Geriatrics and Welfare Research, National Health Research Institutes, 35, Keyan Road, Zhunan, Miaoli County, 350401, Taiwan, 886 037-206-166 ext 51001.
- Department of Psychiatry, National Taiwan University Hospital, Taipei, Taiwan.
- Department of Psychiatry, National Taiwan University Hospital, Yunlin Branch, Douliu, Taiwan.
ORCID-Profile: Chien-Hung Chen, Po-Chang Tseng, Hong-Jie Dai, Chu-Hsien Su, Shi-Heng Wang, Yi-Ling Chien, Wei-Lieh Huang, Chi-Shin Wu, Hsin-Hsi Chen
Zitieren
APA
Chen, C. H., Tseng, P. C., Dai, H. J., Su, C. H., Wang, S. H., Chien, Y. L., Huang, W. L., Wu, C. S., & Chen, H. H. (2026). Two-Stage Extraction of Clinical Course Information From Psychiatric Discharge Summaries Using Fine-Tuned Large Language Models: Cross-Validation Study. JMIR Formative Research, 10, e94454-e94454. https://doi.org/10.2196/94454
Vancouver
Chen CH, Tseng PC, Dai HJ, Su CH, Wang SH, Chien YL, et al. Two-Stage Extraction of Clinical Course Information From Psychiatric Discharge Summaries Using Fine-Tuned Large Language Models: Cross-Validation Study. JMIR Formative Research. 2026;10:e94454-e94454. doi:10.2196/94454
BibTeX
@article{Chen2026,
author = {Chen, Chien-Hung and Tseng, Po-Chang and Dai, Hong-Jie and Su, Chu-Hsien and Wang, Shi-Heng and Chien, Yi-Ling and Huang, Wei-Lieh and Wu, Chi-Shin and Chen, Hsin-Hsi},
title = {Two-Stage Extraction of Clinical Course Information From Psychiatric Discharge Summaries Using Fine-Tuned Large Language Models: Cross-Validation Study},
journal = {JMIR Formative Research},
year = {2026},
volume = {10},
pages = {e94454--e94454},
doi = {10.2196/94454}
}
EUVIMED