EUVIMEDEUVIMED entdecken
Crossref

Performances of Large Language Models in Detecting Psychiatric Diagnoses from Chinese Electronic Medical Records: Comparisons between GPT-3.5, GPT-4, and GPT-4o

Chien Wen Chien, Yueh-Ming Tai

Taiwanese Journal of Psychiatry · 2024-07 · 38 (3) · 134-141

DOI: 10.4103/tpsy.tpsy_25_24 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Taiwanese Journal of Psychiatry, online erschienen am 30.09.2024. Verfasst von Chien Wen Chien und 1 weiteren Autor:innen. Die Arbeit wurde bisher 4-mal zitiert (Crossref) und nennt 38 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

As a type of artificial intelligence (AI), the large language model (LLM) is designed to understand and generate human-like fluent texts. Typical LLMs, e.g., GPT-3.5, GPT-4, and GPT-4o, interact with users through “prompts” and some internal parameters, like “temperature.” Currently, some AI models have been widely used in the field of psychiatry, but systemic reports examining the capacity and suitability of LLM in detecting psychiatry diagnoses are still lacking. In this study, we intended to explore the performances of different generations of LLMs with different levels of temperature in detecting mental illnesses from electronic medical records (EMRs).

Methods

We collected 500 Chinese EMRs from one mental hospital in northern Taiwan, with the “current medical history” section as corpuses. We used the GPT-3.5-turbo-16K, GPT-4, and GPT-4o models provided by Microsoft’s Azure OpenAI service (www.portal.azure.com) to generate AI-based predictions (the probability) for the diagnoses of major depressive disorder (MDD), schizophrenia (SCZ), attention-deficit/hyperactivity disorder (ADHD), and autistic spectrum disorder (ASD). Clinic diagnoses made by qualified psychiatrists were treated as gold standards (target) of receiver operating characteristic curve analysis. Then, their area under the ROC curve (AUCs) were compared using the DeLong test.

Results

Among 500 recruited Chinese EMRs in this study, 56.6% were primarily diagnosed with MDD, as well as 22.4% with SCZ, 11.2% with ADHD, and 9.2% with ASD. In general, our LLMs achieved AUCs of 0.84 to 0.98 for detecting four different diagnoses. There were no significant differences between versions, but newer versions (GPT-4o models with AUCs of 0.98–0.97 for SCZ, ADHD, and ASD) performed better than older versions (GPT-3.5 models with AUCs of 0.88–0.96) except for MDD (AUC of 0.95 for GPT-4 and AUC of 0.93 for GPT-4o). Although DeLong tests showed nonsignificant differences between the AUCs of models with different levels of temperature, models with zero temperatures generally represented the best performances in magnitudes.

Conclusion

To the best of our knowledge, this study is the first to demonstrate that LLMs performed excellently in distinguishing some mental illnesses. Nevertheless, the diagnostic capabilities of LLMs differed from other diagnoses such as MDD. We hypothesize that this phenomenon may partially result from the complexity of symptomology and/or the content filtering rules of OpenAI. Therefore, more advanced models, e.g., GPT-5, or private training models, e.g., Llamma 3, with the relevance generative answering technique, are expected to answer our questions.

Zitieren

APA

Chien, C. W., & Tai, Y. (2024). Performances of Large Language Models in Detecting Psychiatric Diagnoses from Chinese Electronic Medical Records: Comparisons between GPT-3.5, GPT-4, and GPT-4o. Taiwanese Journal of Psychiatry, 38(3), 134-141. https://doi.org/10.4103/tpsy.tpsy_25_24

Vancouver

Chien CW, Tai Y. Performances of Large Language Models in Detecting Psychiatric Diagnoses from Chinese Electronic Medical Records: Comparisons between GPT-3.5, GPT-4, and GPT-4o. Taiwanese Journal of Psychiatry. 2024;38(3):134-141. doi:10.4103/tpsy.tpsy_25_24

BibTeX
@article{Chien2024,
  author = {Chien, Chien Wen and Tai, Yueh-Ming},
  title = {Performances of Large Language Models in Detecting Psychiatric Diagnoses from Chinese Electronic Medical Records: Comparisons between GPT-3.5, GPT-4, and GPT-4o},
  journal = {Taiwanese Journal of Psychiatry},
  year = {2024},
  volume = {38},
  number = {3},
  pages = {134--141},
  doi = {10.4103/tpsy.tpsy_25_24}
}
Kostenlos starten →