Large Language Models for Psychiatric Diagnosis Based on Multicenter Real-World Clinical Records: Comparative Study
JMIR Medical Informatics · 2026-01-13 · 14 · e77699
DOI: 10.2196/77699 ↗At a glance
Multizentrische Studie in JMIR Medical Informatics, published online on 2026-01-13. Written by Maoqian Sun and 8 co-authors. Institution: Mental Health Center, University-Town Hospital of Chongqing Medical University, Chongqing, China. Major MEDLINE topics: Electronic Health Records, Mental Disorders, Language. Cited 5 times so far (Crossref); 33 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Psychiatric disorders are diagnostically challenging and often rely on subjective clinical judgment, particularly in resource-limited settings. Large language models (LLMs) have demonstrated potential in supporting psychiatric diagnosis; however, robust evidence from large-scale, real-world clinical data remains limited.
Objective
This study aimed to evaluate and compare the diagnostic performance of multiple LLMs for psychiatric disorders using multicenter real-world electronic health records (EHRs).
Methods
We retrospectively analyzed 9923 inpatient EHRs collected from 6 psychiatric centers across China, encompassing all ICD-10 (International Statistical Classification of Diseases, Tenth Revision) psychiatric categories. In total, 3 LLMs—GPT-4.0 (OpenAI), GPT-3.5 (OpenAI), and GLM-4-Plus (Zhipu AI)—were evaluated against physician-confirmed discharge diagnoses. Diagnostic performance was assessed using strict accuracy criteria and lenient classification metrics, with subgroup analyses conducted across diagnostic categories and age groups.
Results
GPT-4.0 achieved the highest overall strict diagnostic accuracy (71.7%) and the highest weighted F1-score under lenient evaluation (0.881), particularly for high-prevalence disorders, such as mood disorders and schizophrenia spectrum disorders. Diagnostic performance varied across age groups, with the highest accuracy observed in older adult patients (up to 79.5%) and lower accuracy in adolescents. Across centers, model performance remained stable, with no significant intercenter differences.
Conclusions
LLMs—especially GPT-4.0—demonstrate promising capability in supporting psychiatric diagnosis using real-world EHRs. However, diagnostic performance varies by age group and disorder category. LLMs should be regarded as assistive tools rather than replacements for clinical judgment, and further validation is needed before routine clinical implementation.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Mental Health Center, University-Town Hospital of Chongqing Medical University, Chongqing, China.
- Sichuan Mental Health Center, The Third Hospital of Mianyang, Mianyang, China.
- Third People's Hospital of Panzhihua, Panzhihua, China.
- The Third People's Hospital of Ganzhou, Ganzhou, China.
- The Third People's Hospital of Foshan, Foshan, China.
- Lanzhou Third People's Hospital, Lanzhou, China.
ORCID profiles: Maoqian Sun, Jia Yu, Zhuhong Long, Yun Yang, Tao Xiao, Jiaquan Liang, Jun Feng, Huaili Deng, Guoping Huang
Cite
APA
Sun, M., Yu, J., Long, Z., Yang, Y., Xiao, T., Liang, J., Feng, J., Deng, H., & Huang, G. (2026). Large Language Models for Psychiatric Diagnosis Based on Multicenter Real-World Clinical Records: Comparative Study. JMIR Medical Informatics, 14, e77699. https://doi.org/10.2196/77699
Vancouver
Sun M, Yu J, Long Z, Yang Y, Xiao T, Liang J, et al. Large Language Models for Psychiatric Diagnosis Based on Multicenter Real-World Clinical Records: Comparative Study. JMIR Medical Informatics. 2026;14:e77699. doi:10.2196/77699
BibTeX
@article{Sun2026,
author = {Sun, Maoqian and Yu, Jia and Long, Zhuhong and Yang, Yun and Xiao, Tao and Liang, Jiaquan and Feng, Jun and Deng, Huaili and Huang, Guoping},
title = {Large Language Models for Psychiatric Diagnosis Based on Multicenter Real-World Clinical Records: Comparative Study},
journal = {JMIR Medical Informatics},
year = {2026},
volume = {14},
pages = {e77699},
doi = {10.2196/77699}
}
EUVIMED