EUVIMEDEUVIMED entdecken
Multizentrische StudieVergleichsstudie

Benchmarking 54 large language model configurations for CAD-RADS scoring: open-weight models approach human-level agreement

Veit Sandfort, Davis M Vigneault, Martin J Willemink, Jie Wu, Richard L Hallett, Koen Nieman, Dominik Fleischmann, Domenico Mastrodicasa

The International Journal of Cardiovascular Imaging · 30.07.2026 · 42 (11) · 2159-2172

DOI: 10.1007/s10554-026-03785-8 ↗

Kurz gefasst

Multizentrische Studie in der Zeitschrift The International Journal of Cardiovascular Imaging, online erschienen am 30.07.2026. Verfasst von Veit Sandfort und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Radiology, Stanford University School of Medicine, 300 Pasteur Drive, Stanford, 94305, California, USA. veit.sandfort@gmail.com. Hauptthemen laut MEDLINE: Large Language Models, Computed Tomography Angiography, Coronary Angiography, Coronary Artery Disease. Die Arbeit nennt 17 Literaturangaben.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The Coronary Artery Disease Reporting and Data System (CAD-RADS) standardizes coronary CT angiography (CCTA) reporting, but not all reports contain CAD-RADS classifications. We benchmarked 54 large language model (LLM) configurations across 50 distinct models, including recent proprietary and open-weight reasoning models, for zero-shot CAD-RADS classification. We retrospectively analyzed 500 anonymized CCTA reports from four hospitals across three U.S. regions. Expert cardiovascular radiologists provided the reference standard (human inter-rater $$\kappa =0.89$$ ). Fifty-four model configurations (50 distinct LLMs; four configurable models tested in both thinking and non-thinking modes) spanning Llama 2 7B through recent thinking models (DeepCogito v2, Gemini 3 Pro) processed reports using identical zero-shot prompts. Performance was measured with unweighted Cohen’s $$\kappa$$ . Two LLMs met both pre-specified non-inferiority criteria ( $$\delta =0.10$$ margin and entire 95% CI within the human inter-rater agreement band, $$\kappa =0.804$$ –0.956): Claude 4.6 Opus ( $$\kappa =0.853$$ , 95% CI 0.817–0.889) and the open-weight Gemma 4 31B ( $$\kappa =0.845$$ , 95% CI 0.810–0.882), which ranked second overall. Both met the criteria at the pre-specified $$\delta =0.10$$ margin; at $$\delta =0.05$$ no model qualified. On the reports originally dictated without a CAD-RADS statement (n=343), the top models reached $$\kappa \approx 0.80$$ . Performance declined with longer thinking chains (proxy for case complexity), but thinking-mode outperformed non-thinking mode on matched difficult reports. Gemma 4 31B remained non-inferior at 3-bit quantization and fits on a 24 GB consumer GPU. Current LLMs extract the CAD-RADS stenosis severity category from unstructured CCTA reports with agreement approaching the human inter-rater band, without task-specific training. Our data shows the remarkable rise of open models. A 31B open-weight model matched top proprietary systems and runs on consumer GPUs, enabling privacy-preserving local deployment for clinical data mining. Graphical Abstract

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsComputed Tomography AngiographyCoronary AngiographyCoronary Artery DiseaseBenchmarkingRadiographic Image Interpretation, Computer-AssistedHumansPredictive Value of TestsRetrospective StudiesObserver VariationReproducibility of ResultsUnited StatesSeverity of Illness Index

Stichwörter der Autor:innen

CAD-RADSCoronary CT angiographyLarge language modelsNatural language processingOpen-weight modelsRadiology report classification

Autor:innen und Institutionen

  • Department of Radiology, Stanford University School of Medicine, 300 Pasteur Drive, Stanford, 94305, California, USA. veit.sandfort@gmail.com.
  • Veterans Affairs Health Care System, Palo Alto, California, USA. veit.sandfort@gmail.com.
  • Department of Radiology, Stanford University School of Medicine, 300 Pasteur Drive, Stanford, 94305, California, USA.
  • Segmed, Inc., #810-3790 El Camino Real, Palo Alto, 94306, California, USA.
  • Stanford Cardiovascular Institute, Stanford University, 265 Campus Drive, Stanford, 94305, California, USA.
  • Department of Radiology, University of Washington School of Medicine, 1959 NE Pacific St, Seattle, 98195, Washington, USA.

Zitieren

APA

Sandfort, V., Vigneault, D. M., Willemink, M. J., Wu, J., Hallett, R. L., Nieman, K., Fleischmann, D., & Mastrodicasa, D. (2026). Benchmarking 54 large language model configurations for CAD-RADS scoring: open-weight models approach human-level agreement. The International Journal of Cardiovascular Imaging, 42(11), 2159-2172. https://doi.org/10.1007/s10554-026-03785-8

Vancouver

Sandfort V, Vigneault DM, Willemink MJ, Wu J, Hallett RL, Nieman K, et al. Benchmarking 54 large language model configurations for CAD-RADS scoring: open-weight models approach human-level agreement. The International Journal of Cardiovascular Imaging. 2026;42(11):2159-2172. doi:10.1007/s10554-026-03785-8

BibTeX
@article{Sandfort2026,
  author = {Sandfort, Veit and Vigneault, Davis M and Willemink, Martin J and Wu, Jie and Hallett, Richard L and Nieman, Koen and Fleischmann, Dominik and Mastrodicasa, Domenico},
  title = {Benchmarking 54 large language model configurations for CAD-RADS scoring: open-weight models approach human-level agreement},
  journal = {The International Journal of Cardiovascular Imaging},
  year = {2026},
  volume = {42},
  number = {11},
  pages = {2159--2172},
  doi = {10.1007/s10554-026-03785-8}
}
Kostenlos starten →