Performance of Open-Source Large Language Models in Psychiatry: Usability Study Through Comparative Analysis of Non-English Records and English Translations
Journal of Medical Internet Research · 18.08.2025 · 27 · e69857-e69857
DOI: 10.2196/69857 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 18.08.2025. Verfasst von Min-Gyu Kim und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Ajou University School of Medicine, 206 World cup-ro, Yeongtong-gu, Suwon, 16499, Republic of Korea, 82 3121944…. Hauptthemen laut MEDLINE: Language, Psychiatry, Translating, Mental Disorders. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 33 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Large language models (LLMs) have emerged as promising tools for addressing global disparities in mental health care. However, cloud-based proprietary models raise concerns about data privacy and limited adaptability to local health care systems. In contrast, open-source LLMs offer several advantages, including enhanced data security, the ability to operate offline in resource-limited settings, and greater adaptability to non-English clinical environments. Nevertheless, their performance in psychiatric applications involving non-English language inputs remains largely unexplored.
Objective
This study aimed to systematically evaluate the clinical reasoning capabilities and diagnostic accuracy of a locally deployable open-source LLM in both Korean and English psychiatric contexts.
Methods
The openbuddy-mistral-7b-v13.1 model, fine-tuned from Mistral 7B to enable conversational capabilities in Korean, was selected. A total of 200 deidentified psychiatric interview notes, documented during initial assessments of emergency department patients, were randomly selected from the electronic medical records of a tertiary hospital in South Korea. The dataset included 50 cases each of schizophrenia, bipolar disorder, depressive disorder, and anxiety disorder. The model translated the Korean notes into English and was prompted to extract 5 clinically meaningful diagnostic clues and generate the 2 most likely diagnoses using both the original Korean and translated English inputs. The hallucination rate and clinical relevance of the generated clues were manually evaluated. Top-1 and top-2 diagnostic accuracy were assessed by comparing the model’s prediction with the ground truth labels. Additionally, the model’s performance on a structured diagnostic task was evaluated using the psychiatry section of the Korean Medical Licensing Examination and its English-translated version.
Results
The model generated 997 clues from Korean interview notes and 1003 clues from English-translated notes. Hallucinations were more frequent with Korean input (n=301, 30.2%) than with English (n=134, 13.4%). Diagnostic relevance was also higher in English (n=429, 42.8%) compared to Korean (n=341, 34.2%). The model showed significantly higher top-1 diagnostic accuracy with English input (74.5% vs 59%; P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Informatics, Ajou University School of Medicine, 206 World cup-ro, Yeongtong-gu, Suwon, 16499, Republic of Korea, 82 312194471, 82 312194472.
- Center for Biomedical Informatics Research, Ajou University Medical Cencer, Suown, Republic of Korea.
- Department of Medical Sciences, Graduate School of Ajou University, Suwon, Korea.
- BK21 R&E Initiative for Advanced Precision Medicine, Suwon, Republic of Korea.
- Department of Biomedical Sciences, Ajou University Graduate School of Medicine, Suwon, Republic of Korea.
- Department of Computer Science, The University of Texas at Austin, Austin, TX, United States.
Zitieren
APA
Kim, M. G., Hwang, G., Chang, J., Chang, S., Roh, H. W., & Park, R. W. (2025). Performance of Open-Source Large Language Models in Psychiatry: Usability Study Through Comparative Analysis of Non-English Records and English Translations. Journal of Medical Internet Research, 27, e69857-e69857. https://doi.org/10.2196/69857
Vancouver
Kim MG, Hwang G, Chang J, Chang S, Roh HW, Park RW. Performance of Open-Source Large Language Models in Psychiatry: Usability Study Through Comparative Analysis of Non-English Records and English Translations. Journal of Medical Internet Research. 2025;27:e69857-e69857. doi:10.2196/69857
BibTeX
@article{Kim2025,
author = {Kim, Min-Gyu and Hwang, Gyubeom and Chang, Junhyuk and Chang, Seheon and Roh, Hyun Woong and Park, Rae Woong},
title = {Performance of Open-Source Large Language Models in Psychiatry: Usability Study Through Comparative Analysis of Non-English Records and English Translations},
journal = {Journal of Medical Internet Research},
year = {2025},
volume = {27},
pages = {e69857--e69857},
doi = {10.2196/69857}
}
EUVIMED