EUVIMEDEUVIMED entdecken
Open Access

Simulated Reasoning and Self-Verification for Psychiatric Diagnosis in Generalist Large Language Models: Comparative Evaluation

Karthik V Sarma, Kaitlin E Hanss, Andrew J M Halls, Daniel F Becker, Anne L Glowinski, Andrew D Krystal

JMIR AI · 08.06.2026 · 5 · e83927-e83927

DOI: 10.2196/83927 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR AI, online erschienen am 08.06.2026. Verfasst von Karthik V Sarma und 5 weiteren Autor:innen. Beteiligte Einrichtung: Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, 675 18th Street, Box 3134, San Francisco, CA, 94107, Unite…. Die Arbeit nennt 24 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIMH NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) and, more recently, large reasoning models (LRMs) have rapidly garnered significant interest for application in psychiatry and behavioral health. However, recent studies have identified significant shortcomings and potential risks in the performance of LLM-based systems, complicating their application to psychiatric diagnosis. Two promising approaches to addressing these challenges and improving the efficacy of these models are simulated reasoning (SR) and self-verification (SV), in which additional “reasoning tokens” are used to guide model output, either during or after inference.

Objective

We aimed to explore how the use of SR (via LRMs) and SV (via supplemental prompting) affects the psychiatric diagnostic performance of LLMs.

Methods

106 case vignettes and associated diagnoses were extracted from the DSM-5-TR (Diagnostic and Statistical Manual, Version 5, Text Revision) Clinical Cases book, with permission. Both an LLM and an LRM model were selected from the latest available model generation for each of the two vendors studied (OpenAI and Google). Two inference approaches were developed: a Basic approach that directly prompted models to provide diagnoses and a SV approach that augmented the Basic approach with additional prompts. All case vignettes were processed by the two LLMs, two LRMs, and two inference approaches, and diagnostic performance was evaluated using the sensitivity and positive predictive value (PPV). Binomial generalized linear mixed models were used to test for significant differences between the model vendors (OpenAI, Google), type (LLM, LRM), and the addition of an SV prompt.

Results

All vignettes were successfully processed by each model and inference approach. Sensitivity ranged from 0.732 to 0.817, and PPV ranged from 0.534 to 0.779. The best overall performance was found in the o3-pro LRM using SV, with a sensitivity of 0.782 and a PPV of 0.779. No statistically significant fixed effects were found for sensitivity. For PPV, a statistically significant effect was found for prompt type (SV, P =.007) and model type (LRM, P =.009). No significant interaction effects were identified.

Conclusions

We found that both SR and SV yielded statistically significant improvements in the PPV, without significant differences in the sensitivity. The addition of the manually specified SV prompt improved the PPV even when simulated reasoning was used. This suggests that future efforts to apply language models in behavioral health could benefit from manually crafted reasoning prompts and automated SR.

Schlagwörter

Stichwörter der Autor:innen

DSMDiagnostic and Statistical Manualdiagnosislarge language modelslarge reasoning modelspsychiatryreasoningself verificationsimulated reasoning

Autor:innen und Institutionen

  • Department of Psychiatry and Behavioral Sciences, University of California, San Francisco, 675 18th Street, Box 3134, San Francisco, CA, 94107, United States, 1 415-476-7527.
  • Philip R Lee Institute for Health Policy Studies, University of California, San Francisco, San Francisco, CA, United States.

Förderung

  • NIMH NIH HHS

Zitieren

APA

Sarma, K. V., Hanss, K. E., Halls, A. J. M., Becker, D. F., Glowinski, A. L., & Krystal, A. D. (2026). Simulated Reasoning and Self-Verification for Psychiatric Diagnosis in Generalist Large Language Models: Comparative Evaluation. JMIR AI, 5, e83927-e83927. https://doi.org/10.2196/83927

Vancouver

Sarma KV, Hanss KE, Halls AJM, Becker DF, Glowinski AL, Krystal AD. Simulated Reasoning and Self-Verification for Psychiatric Diagnosis in Generalist Large Language Models: Comparative Evaluation. JMIR AI. 2026;5:e83927-e83927. doi:10.2196/83927

BibTeX
@article{Sarma2026,
  author = {Sarma, Karthik V and Hanss, Kaitlin E and Halls, Andrew J M and Becker, Daniel F and Glowinski, Anne L and Krystal, Andrew D},
  title = {Simulated Reasoning and Self-Verification for Psychiatric Diagnosis in Generalist Large Language Models: Comparative Evaluation},
  journal = {JMIR AI},
  year = {2026},
  volume = {5},
  pages = {e83927--e83927},
  doi = {10.2196/83927}
}
Kostenlos starten →