EUVIMEDEUVIMED entdecken
Open Access

Two-Stage Extraction of Clinical Course Information From Psychiatric Discharge Summaries Using Fine-Tuned Large Language Models: Cross-Validation Study

Chien-Hung Chen iD, Po-Chang Tseng iD, Hong-Jie Dai iD, Chu-Hsien Su iD, Shi-Heng Wang iD, Yi-Ling Chien iD, Wei-Lieh Huang iD, Chi-Shin Wu iD, Hsin-Hsi Chen iD

JMIR Formative Research · 06.08.2026 · 10 · e94454-e94454

DOI: 10.2196/94454 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Formative Research, online erschienen am 06.08.2026. Verfasst von Chien-Hung Chen und 8 weiteren Autor:innen. Beteiligte Einrichtung: Graduate Institute of Networking and Multimedia, National Taiwan University, Taipei, Taiwan. Hauptthemen laut MEDLINE: Large Language Models, Patient Discharge Summaries, Patient Discharge, Mental Disorders. Die Arbeit nennt 20 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Clinical course information, including disease onset, episode recurrence, and hospitalization history, is essential for psychiatric care and research. However, these data are embedded in unstructured clinical narratives with substantial linguistic variability, making manual extraction labor-intensive and rule-based extraction difficult to scale. Fine-tuned large language models (LLMs) may flexibly extract such information from privacy-sensitive psychiatric records.

Objective

This study aimed to evaluate the performance of LLMs for automatically extracting temporal and clinical course information from psychiatric discharge summaries.

Methods

We analyzed 500 psychiatric discharge summaries from the Integrated Medical Database of National Taiwan University Hospital. A psychiatrist and a natural language processing researcher manually annotated clinical events and temporal information. Four open-source LLMs (LLaMA, MentaLLaMA, OpenBioLLM, and Mistral) were fine-tuned using low-rank adaptation and evaluated using 10-fold cross-validation. A 2-stage framework was developed in which sentence-level extraction of clinical events and temporal information was followed by chart-level prediction of 4 clinical course features: first-episode onset time, episode count, number of psychiatric hospitalizations, and most recent hospitalization. Performance was assessed using precision, recall, F 1 -score, accuracy, mean absolute error (MAE), and bootstrap significance testing.

Results

A total of 12,947 sentences were extracted from 500 discharge summaries, yielding 7177 clinical event annotations and 4842 temporal annotations. At the sentence level, Mistral achieved the highest F 1 -scores for clinical event extraction, including symptom/episode detection (0.925, 95% CI 0.920‐0.930), hospitalization detection (0.944, 95% CI 0.934‐0.952), and remission/response detection (0.867, 95% CI 0.851‐0.883). Mistral also achieved the highest F 1 -scores for most temporal information categories, including age expressions (0.983, 95% CI 0.973‐0.993), relative time expressions (0.953, 95% CI 0.944‐0.963), duration expressions (0.976, 95% CI 0.962‐0.987), and vague temporal expressions (0.888, 95% CI 0.871‐0.905). At the chart level, the proposed 2-stage framework showed the clearest benefit for first-episode onset prediction. Using the 2-stage framework, Mistral achieved the highest point estimate for onset accuracy (0.772), although its performance did not significantly differ from that of LLaMA using the same framework (0.744; bootstrap P =.26). For onset prediction, the 2-stage framework significantly outperformed the direct and joint extraction approaches across all evaluated models (all bootstrap P ≤.002). For other chart-level features, the best-performing approach varied: using the 2-stage framework, Mistral achieved the highest episode count accuracy (0.624; MAE=0.428); using the direct approach, LLaMA achieved the highest hospitalization count accuracy (0.692; MAE=0.379); and using the 2-stage framework, OpenBioLLM achieved the highest most recent hospitalization accuracy (0.868; F 1 -score=0.617).

Conclusions

Fine-tuned, locally deployable, open-source LLMs can extract temporal and longitudinal disease course information from psychiatric discharge summaries. The 2-stage framework was most beneficial for first-episode onset prediction and performed competitively across other chart-level features, supporting the use of LLMs to transform heterogeneous psychiatric narratives into structured data for research and decision support.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsPatient Discharge SummariesPatient DischargeMental DisordersHumansTaiwanNatural Language Processing

Stichwörter der Autor:innen

medical record linkagemedical recordsmental disordersnatural language processingtime factors

Autor:innen und Institutionen

  • Graduate Institute of Networking and Multimedia, National Taiwan University, Taipei, Taiwan.
  • Institute of Psychiatry, Psychology & Neuroscience, King's College London, London, United Kingdom.
  • Department of Electrical Engineering, College of Electrical Engineering and Computer Science, National Kaohsiung University of Science and Technology, Kaohsiung, Taiwan.
  • National Center for Geriatrics and Welfare Research, National Health Research Institutes, 35, Keyan Road, Zhunan, Miaoli County, 350401, Taiwan, 886 037-206-166 ext 51001.
  • Department of Psychiatry, National Taiwan University Hospital, Taipei, Taiwan.
  • Department of Psychiatry, National Taiwan University Hospital, Yunlin Branch, Douliu, Taiwan.

ORCID-Profile: Chien-Hung Chen, Po-Chang Tseng, Hong-Jie Dai, Chu-Hsien Su, Shi-Heng Wang, Yi-Ling Chien, Wei-Lieh Huang, Chi-Shin Wu, Hsin-Hsi Chen

Zitieren

APA

Chen, C. H., Tseng, P. C., Dai, H. J., Su, C. H., Wang, S. H., Chien, Y. L., Huang, W. L., Wu, C. S., & Chen, H. H. (2026). Two-Stage Extraction of Clinical Course Information From Psychiatric Discharge Summaries Using Fine-Tuned Large Language Models: Cross-Validation Study. JMIR Formative Research, 10, e94454-e94454. https://doi.org/10.2196/94454

Vancouver

Chen CH, Tseng PC, Dai HJ, Su CH, Wang SH, Chien YL, et al. Two-Stage Extraction of Clinical Course Information From Psychiatric Discharge Summaries Using Fine-Tuned Large Language Models: Cross-Validation Study. JMIR Formative Research. 2026;10:e94454-e94454. doi:10.2196/94454

BibTeX
@article{Chen2026,
  author = {Chen, Chien-Hung and Tseng, Po-Chang and Dai, Hong-Jie and Su, Chu-Hsien and Wang, Shi-Heng and Chien, Yi-Ling and Huang, Wei-Lieh and Wu, Chi-Shin and Chen, Hsin-Hsi},
  title = {Two-Stage Extraction of Clinical Course Information From Psychiatric Discharge Summaries Using Fine-Tuned Large Language Models: Cross-Validation Study},
  journal = {JMIR Formative Research},
  year = {2026},
  volume = {10},
  pages = {e94454--e94454},
  doi = {10.2196/94454}
}
Kostenlos starten →