EUVIMEDOpdag EUVIMED
Open access

Two-Stage Extraction of Clinical Course Information From Psychiatric Discharge Summaries Using Fine-Tuned Large Language Models: Cross-Validation Study

Chien-Hung Chen iD, Po-Chang Tseng iD, Hong-Jie Dai iD, Chu-Hsien Su iD, Shi-Heng Wang iD, Yi-Ling Chien iD, Wei-Lieh Huang iD, Chi-Shin Wu iD, Hsin-Hsi Chen iD

JMIR Formative Research · 2026-08-06 · 10 · e94454-e94454

DOI: 10.2196/94454 ↗

At a glance

Journal article in JMIR Formative Research, published online on 2026-08-06. Written by Chien-Hung Chen and 8 co-authors. Institution: Graduate Institute of Networking and Multimedia, National Taiwan University, Taipei, Taiwan. Major MEDLINE topics: Large Language Models, Patient Discharge Summaries, Patient Discharge, Mental Disorders. 20 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Clinical course information, including disease onset, episode recurrence, and hospitalization history, is essential for psychiatric care and research. However, these data are embedded in unstructured clinical narratives with substantial linguistic variability, making manual extraction labor-intensive and rule-based extraction difficult to scale. Fine-tuned large language models (LLMs) may flexibly extract such information from privacy-sensitive psychiatric records.

Objective

This study aimed to evaluate the performance of LLMs for automatically extracting temporal and clinical course information from psychiatric discharge summaries.

Methods

We analyzed 500 psychiatric discharge summaries from the Integrated Medical Database of National Taiwan University Hospital. A psychiatrist and a natural language processing researcher manually annotated clinical events and temporal information. Four open-source LLMs (LLaMA, MentaLLaMA, OpenBioLLM, and Mistral) were fine-tuned using low-rank adaptation and evaluated using 10-fold cross-validation. A 2-stage framework was developed in which sentence-level extraction of clinical events and temporal information was followed by chart-level prediction of 4 clinical course features: first-episode onset time, episode count, number of psychiatric hospitalizations, and most recent hospitalization. Performance was assessed using precision, recall, F 1 -score, accuracy, mean absolute error (MAE), and bootstrap significance testing.

Results

A total of 12,947 sentences were extracted from 500 discharge summaries, yielding 7177 clinical event annotations and 4842 temporal annotations. At the sentence level, Mistral achieved the highest F 1 -scores for clinical event extraction, including symptom/episode detection (0.925, 95% CI 0.920‐0.930), hospitalization detection (0.944, 95% CI 0.934‐0.952), and remission/response detection (0.867, 95% CI 0.851‐0.883). Mistral also achieved the highest F 1 -scores for most temporal information categories, including age expressions (0.983, 95% CI 0.973‐0.993), relative time expressions (0.953, 95% CI 0.944‐0.963), duration expressions (0.976, 95% CI 0.962‐0.987), and vague temporal expressions (0.888, 95% CI 0.871‐0.905). At the chart level, the proposed 2-stage framework showed the clearest benefit for first-episode onset prediction. Using the 2-stage framework, Mistral achieved the highest point estimate for onset accuracy (0.772), although its performance did not significantly differ from that of LLaMA using the same framework (0.744; bootstrap P =.26). For onset prediction, the 2-stage framework significantly outperformed the direct and joint extraction approaches across all evaluated models (all bootstrap P ≤.002). For other chart-level features, the best-performing approach varied: using the 2-stage framework, Mistral achieved the highest episode count accuracy (0.624; MAE=0.428); using the direct approach, LLaMA achieved the highest hospitalization count accuracy (0.692; MAE=0.379); and using the 2-stage framework, OpenBioLLM achieved the highest most recent hospitalization accuracy (0.868; F 1 -score=0.617).

Conclusions

Fine-tuned, locally deployable, open-source LLMs can extract temporal and longitudinal disease course information from psychiatric discharge summaries. The 2-stage framework was most beneficial for first-episode onset prediction and performed competitively across other chart-level features, supporting the use of LLMs to transform heterogeneous psychiatric narratives into structured data for research and decision support.

Subjects

MeSH terms (MEDLINE)

Large Language ModelsPatient Discharge SummariesPatient DischargeMental DisordersHumansTaiwanNatural Language Processing

Author keywords

medical record linkagemedical recordsmental disordersnatural language processingtime factors

Authors and institutions

  • Graduate Institute of Networking and Multimedia, National Taiwan University, Taipei, Taiwan.
  • Institute of Psychiatry, Psychology & Neuroscience, King's College London, London, United Kingdom.
  • Department of Electrical Engineering, College of Electrical Engineering and Computer Science, National Kaohsiung University of Science and Technology, Kaohsiung, Taiwan.
  • National Center for Geriatrics and Welfare Research, National Health Research Institutes, 35, Keyan Road, Zhunan, Miaoli County, 350401, Taiwan, 886 037-206-166 ext 51001.
  • Department of Psychiatry, National Taiwan University Hospital, Taipei, Taiwan.
  • Department of Psychiatry, National Taiwan University Hospital, Yunlin Branch, Douliu, Taiwan.

ORCID profiles: Chien-Hung Chen, Po-Chang Tseng, Hong-Jie Dai, Chu-Hsien Su, Shi-Heng Wang, Yi-Ling Chien, Wei-Lieh Huang, Chi-Shin Wu, Hsin-Hsi Chen

Cite

APA

Chen, C. H., Tseng, P. C., Dai, H. J., Su, C. H., Wang, S. H., Chien, Y. L., Huang, W. L., Wu, C. S., & Chen, H. H. (2026). Two-Stage Extraction of Clinical Course Information From Psychiatric Discharge Summaries Using Fine-Tuned Large Language Models: Cross-Validation Study. JMIR Formative Research, 10, e94454-e94454. https://doi.org/10.2196/94454

Vancouver

Chen CH, Tseng PC, Dai HJ, Su CH, Wang SH, Chien YL, et al. Two-Stage Extraction of Clinical Course Information From Psychiatric Discharge Summaries Using Fine-Tuned Large Language Models: Cross-Validation Study. JMIR Formative Research. 2026;10:e94454-e94454. doi:10.2196/94454

BibTeX
@article{Chen2026,
  author = {Chen, Chien-Hung and Tseng, Po-Chang and Dai, Hong-Jie and Su, Chu-Hsien and Wang, Shi-Heng and Chien, Yi-Ling and Huang, Wei-Lieh and Wu, Chi-Shin and Chen, Hsin-Hsi},
  title = {Two-Stage Extraction of Clinical Course Information From Psychiatric Discharge Summaries Using Fine-Tuned Large Language Models: Cross-Validation Study},
  journal = {JMIR Formative Research},
  year = {2026},
  volume = {10},
  pages = {e94454--e94454},
  doi = {10.2196/94454}
}
Start for free →