EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Evaluating Large Language Models for Psychological Diagnosis and Counseling: A Dual-Task Framework with Cultural Reflection

Aitong He

Scientific Journal of Intelligent Systems Research · 29.06.2026 · 8 (5) · 157-169

DOI: 10.54691/8f2d4016 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Scientific Journal of Intelligent Systems Research, online erschienen am 29.06.2026. Verfasst von Aitong He. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) have demonstrated high potential in healthcare and mental health related applications, however, their applicability to psychological diagnosis and counseling is not adequately investigated. This paper presents a dual-task assessment model that can be used to systematically evaluate how the mainstream Chinese LLMs can be used in psychology and through two lens psychological diagnosis and psychological counseling. In the diagnosis task, we test four Chinese LLMs, DeepSeek, Doubao, Kimi, and Tongyi Qianwen, on a mental health dataset that consists of five psychological disorders. Accuracy, Precision, Recall, F1-score, AUC, and confusion matrices are the measures of their diagnostic performance. In the case of the counseling task, we develop a prompt-based assessment system and evaluate the generated responses using six dimensions, including Overall, Empathy, Specificity, Medical Advice, Factual Consistency and Toxicity. To lower the expenses of annotating on the large scale with experts, we use an LLM-as-a-Judge approach to rank and evaluate the generated counseling answers. The experimental findings indicate that already existing Chinese LLM has some promising prospects in preliminary psychological diagnosis but fails to differentiate semantically similar disorders resulting in diagnostic confusion. The models themselves tend to produce fluent and empathetic responses in a counseling situation although they display obvious limitations in specificity, safety awareness, and dealing with medical advice. In addition, we also offer a case analysis and cultural reflection, pointing out that good average performance does not always imply the consistent support of culturally diverse adolescent groups. On the whole, our results indicate that the existing LLMs are more appropriate as an assistant, rather than a replacement of a mental health professional, and the research of the future should be aimed at enhancing the safety, cultural sensitivity, and clinical reliability of the system.

Zitieren

APA

He, A. (2026). Evaluating Large Language Models for Psychological Diagnosis and Counseling: A Dual-Task Framework with Cultural Reflection. Scientific Journal of Intelligent Systems Research, 8(5), 157-169. https://doi.org/10.54691/8f2d4016

Vancouver

He A. Evaluating Large Language Models for Psychological Diagnosis and Counseling: A Dual-Task Framework with Cultural Reflection. Scientific Journal of Intelligent Systems Research. 2026;8(5):157-169. doi:10.54691/8f2d4016

BibTeX
@article{He2026,
  author = {He, Aitong},
  title = {Evaluating Large Language Models for Psychological Diagnosis and Counseling: A Dual-Task Framework with Cultural Reflection},
  journal = {Scientific Journal of Intelligent Systems Research},
  year = {2026},
  volume = {8},
  number = {5},
  pages = {157--169},
  doi = {10.54691/8f2d4016}
}
Kostenlos starten →