EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Diagnostic Accuracy of GPT‐Based Large Language Models Across Versions, Prompting Techniques, and Case Presentation Formats

Seraphina Fong, Alessandro Carollo, Martina Dal Maso, Giovanni Martinotti, Debora Luciani, Yasser Saeed Khan, Luca Pellegrini, Ornella Corazza, Gianluca Esposito

Human Behavior and Emerging Technologies · 2026-01 · 2026 (1)

DOI: 10.1155/hbe2/4674484 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Human Behavior and Emerging Technologies, online erschienen am 21.04.2026. Verfasst von Seraphina Fong und 8 weiteren Autor:innen. Die Arbeit nennt 45 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Università degli Studi di Trento.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Despite the centrality of the diagnostic assessment in psychiatry, the agreement among mental health practitioners often varies from poor to moderate. The potential of large language models (LLMs; such as gpt‐based models), among other approaches, has been studied to be used as standardized tools to support clinicians’ decision‐making. The current work investigates the diagnostic accuracy of gpt‐based LLMs (gpt‐3.5 and gpt‐5.1) across different case presentation styles (i.e., vignette and outline) and prompting techniques. A total of 46 psychiatric cases with an accompanying diagnosis were used. Two trained clinical psychologists evaluated the proximity of the generated diagnosis against the reference diagnosis. A robust statistical approach was then used to investigate the effect of case format and prompt type on the average diagnostic accuracy. Importantly, accuracy in this context reflects alignment with a reference label under constrained vignette‐based inputs, rather than equivalence with comprehensive clinical diagnostic practice. The results showed a strong agreement between the ratings of the two clinical psychologists ( kappa = 0.798), with moderate agreement for gpt‐3.5’s diagnoses and almost perfect for gpt‐5.1’s diagnoses. Overall, gpt‐5.1 showed higher diagnostic accuracy and proximity to human diagnostic evaluations than gpt‐3.5 ( p < 0.001). For gpt‐3.5, a small but statistically significant main effect of prompting technique on diagnostic accuracy emerged ( p = 0.009). The highest proximity to the reference diagnosis was achieved when gpt‐3.5 was simply instructed to provide and justify a single diagnosis for each case, as compared to when it was asked to provide a diagnosis likelihood ( p < 0.001) or when it was asked to act as a clinical psychologist ( p = 0.001). Conversely, gpt‐5.1 showed high performance independent of the prompting technique and case format. Under these experimental conditions, the results of the current work provide preliminary evidence supporting the potential use of LLMs as tools to assist the diagnostic process in psychiatry and provide general indication for slightly optimizing their performance. Additionally, this study offers a methodological framework that can serve as an example for future research aiming to systematically evaluate LLMs’ diagnostic capabilities across different prompting strategies and case presentation formats.

Förderung

  • Università degli Studi di Trento

Zitieren

APA

Fong, S., Carollo, A., Maso, M. D., Martinotti, G., Luciani, D., Khan, Y. S., Pellegrini, L., Corazza, O., & Esposito, G. (2026). Diagnostic Accuracy of GPT‐Based Large Language Models Across Versions, Prompting Techniques, and Case Presentation Formats. Human Behavior and Emerging Technologies, 2026(1). https://doi.org/10.1155/hbe2/4674484

Vancouver

Fong S, Carollo A, Maso MD, Martinotti G, Luciani D, Khan YS, et al. Diagnostic Accuracy of GPT‐Based Large Language Models Across Versions, Prompting Techniques, and Case Presentation Formats. Human Behavior and Emerging Technologies. 2026;2026(1). doi:10.1155/hbe2/4674484

BibTeX
@article{Fong2026,
  author = {Fong, Seraphina and Carollo, Alessandro and Maso, Martina Dal and Martinotti, Giovanni and Luciani, Debora and Khan, Yasser Saeed and Pellegrini, Luca and Corazza, Ornella and Esposito, Gianluca},
  title = {Diagnostic Accuracy of GPT‐Based Large Language Models Across Versions, Prompting Techniques, and Case Presentation Formats},
  journal = {Human Behavior and Emerging Technologies},
  year = {2026},
  volume = {2026},
  number = {1},
  doi = {10.1155/hbe2/4674484}
}
Kostenlos starten →