EUVIMEDDécouvrir EUVIMED
Multizentrische StudieVergleichsstudieOpen access

Large Language Models for Psychiatric Diagnosis Based on Multicenter Real-World Clinical Records: Comparative Study

Maoqian Sun iD, Jia Yu iD, Zhuhong Long iD, Yun Yang iD, Tao Xiao iD, Jiaquan Liang iD, Jun Feng iD, Huaili Deng iD, Guoping Huang iD

JMIR Medical Informatics · 2026-01-13 · 14 · e77699

DOI: 10.2196/77699 ↗

At a glance

Multizentrische Studie in JMIR Medical Informatics, published online on 2026-01-13. Written by Maoqian Sun and 8 co-authors. Institution: Mental Health Center, University-Town Hospital of Chongqing Medical University, Chongqing, China. Major MEDLINE topics: Electronic Health Records, Mental Disorders, Language. Cited 5 times so far (Crossref); 33 references. The full text is freely available in PubMed Central.

RésuméSubjectsRéférences bibliographiquesCiteRelated studies

Abstract (original language)

Background

Psychiatric disorders are diagnostically challenging and often rely on subjective clinical judgment, particularly in resource-limited settings. Large language models (LLMs) have demonstrated potential in supporting psychiatric diagnosis; however, robust evidence from large-scale, real-world clinical data remains limited.

Objective

This study aimed to evaluate and compare the diagnostic performance of multiple LLMs for psychiatric disorders using multicenter real-world electronic health records (EHRs).

Methods

We retrospectively analyzed 9923 inpatient EHRs collected from 6 psychiatric centers across China, encompassing all ICD-10 (International Statistical Classification of Diseases, Tenth Revision) psychiatric categories. In total, 3 LLMs—GPT-4.0 (OpenAI), GPT-3.5 (OpenAI), and GLM-4-Plus (Zhipu AI)—were evaluated against physician-confirmed discharge diagnoses. Diagnostic performance was assessed using strict accuracy criteria and lenient classification metrics, with subgroup analyses conducted across diagnostic categories and age groups.

Results

GPT-4.0 achieved the highest overall strict diagnostic accuracy (71.7%) and the highest weighted F1-score under lenient evaluation (0.881), particularly for high-prevalence disorders, such as mood disorders and schizophrenia spectrum disorders. Diagnostic performance varied across age groups, with the highest accuracy observed in older adult patients (up to 79.5%) and lower accuracy in adolescents. Across centers, model performance remained stable, with no significant intercenter differences.

Conclusions

LLMs—especially GPT-4.0—demonstrate promising capability in supporting psychiatric diagnosis using real-world EHRs. However, diagnostic performance varies by age group and disorder category. LLMs should be regarded as assistive tools rather than replacements for clinical judgment, and further validation is needed before routine clinical implementation.

Subjects

MeSH terms (MEDLINE)

Electronic Health RecordsMental DisordersLanguageHumansRetrospective StudiesMaleFemaleAdultMiddle AgedChinaAdolescentInternational Classification of DiseasesYoung AdultAgedLarge Language Models

Author keywords

artificial intelligencelarge language modelsmulticenter studypsychiatric disordersreal-world data

Authors and institutions

  • Mental Health Center, University-Town Hospital of Chongqing Medical University, Chongqing, China.
  • Sichuan Mental Health Center, The Third Hospital of Mianyang, Mianyang, China.
  • Third People's Hospital of Panzhihua, Panzhihua, China.
  • The Third People's Hospital of Ganzhou, Ganzhou, China.
  • The Third People's Hospital of Foshan, Foshan, China.
  • Lanzhou Third People's Hospital, Lanzhou, China.

ORCID profiles: Maoqian Sun, Jia Yu, Zhuhong Long, Yun Yang, Tao Xiao, Jiaquan Liang, Jun Feng, Huaili Deng, Guoping Huang

Cite

APA

Sun, M., Yu, J., Long, Z., Yang, Y., Xiao, T., Liang, J., Feng, J., Deng, H., & Huang, G. (2026). Large Language Models for Psychiatric Diagnosis Based on Multicenter Real-World Clinical Records: Comparative Study. JMIR Medical Informatics, 14, e77699. https://doi.org/10.2196/77699

Vancouver

Sun M, Yu J, Long Z, Yang Y, Xiao T, Liang J, et al. Large Language Models for Psychiatric Diagnosis Based on Multicenter Real-World Clinical Records: Comparative Study. JMIR Medical Informatics. 2026;14:e77699. doi:10.2196/77699

BibTeX
@article{Sun2026,
  author = {Sun, Maoqian and Yu, Jia and Long, Zhuhong and Yang, Yun and Xiao, Tao and Liang, Jiaquan and Feng, Jun and Deng, Huaili and Huang, Guoping},
  title = {Large Language Models for Psychiatric Diagnosis Based on Multicenter Real-World Clinical Records: Comparative Study},
  journal = {JMIR Medical Informatics},
  year = {2026},
  volume = {14},
  pages = {e77699},
  doi = {10.2196/77699}
}
Start for free →