LLA MADRS: Evaluating Open-Source LLMs on Real Clinical Interviews—To Reason or Not to Reason?
Transactions of the Association for Computational Linguistics · 01.07.2026 · 14 · 1568-1583
DOI: 10.1162/tacl.a.741 ↗Kurz gefasst
Fachartikel in der Zeitschrift Transactions of the Association for Computational Linguistics, online erschienen am 01.07.2026. Verfasst von Gaoussou Youssouf Kebe und 5 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large language models (LLMs) excel on many NLP benchmarks, but their behavior on real-world, semi-structured prediction remains underexplored. We present LLAMADRS, a benchmark for structured clinical assessment from dialogue built on the CAMI corpus of psychiatric interviews, comprising 5,804 expert annotations across 541 sessions. We evaluate 25 open-source models (standard and reasoning-augmented; 0.6B–400B parameters) and generate over 400,000 predictions. Our results demonstrate that strong open-source LLMs achieve item-level accuracy with residual error below clinically substantial thresholds. Additionally, an Item-then-Sum (ITS) strategy, assessing symptoms individually through discrete LLM calls before synthesizing final scores, significantly reduces error relative to Direct Total Score (DTS) prediction across most model architectures and scales, despite reasoning models attempting similar decomposition in the reasoning traces of their DTS predictions. In fact, we find that performance gains attributed to “reasoning” depend fundamentally on prompt design: standard models equipped with structured task definitions and examples match reasoning-augmented counterparts. Among the latter, longer reasoning traces correlate with reduced error; while higher model scale does across both architectures. Our results clarify when and why reasoning helps and offer actionable guidance for deploying LLMs in semi-structured clinical assessment.
Zitieren
APA
Kebe, G. Y., Girard, J. M., Liebenthal, E., Baker, J., Torre, F. D. L., & Morency, L. (2026). LLA MADRS: Evaluating Open-Source LLMs on Real Clinical Interviews—To Reason or Not to Reason?. Transactions of the Association for Computational Linguistics, 14, 1568-1583. https://doi.org/10.1162/tacl.a.741
Vancouver
Kebe GY, Girard JM, Liebenthal E, Baker J, Torre FDL, Morency L. LLA MADRS: Evaluating Open-Source LLMs on Real Clinical Interviews—To Reason or Not to Reason?. Transactions of the Association for Computational Linguistics. 2026;14:1568-1583. doi:10.1162/tacl.a.741
BibTeX
@article{Kebe2026,
author = {Kebe, Gaoussou Youssouf and Girard, Jeffrey M. and Liebenthal, Einat and Baker, Justin and Torre, Fernando De la and Morency, Louis-Philippe},
title = {LLA MADRS: Evaluating Open-Source LLMs on Real Clinical Interviews—To Reason or Not to Reason?},
journal = {Transactions of the Association for Computational Linguistics},
year = {2026},
volume = {14},
pages = {1568--1583},
doi = {10.1162/tacl.a.741}
}
EUVIMED