Large Language Models for CAD-RADS 2.0 Extraction From Semi-Structured Coronary CT Angiography Reports: A Multi-Institutional Study
Korean Journal of Radiology · 2025 · 26 (9) · 817
DOI: 10.3348/kjr.2025.0293 ↗Kurz gefasst
Multizentrische Studie in der Zeitschrift Korean Journal of Radiology, 2025. Verfasst von Dabin Min und 10 weiteren Autor:innen. Beteiligte Einrichtung: Interdisciplinary Program in Bioengineering, Seoul National University Graduate School, Seoul, Republic of Korea. Hauptthemen laut MEDLINE: Computed Tomography Angiography, Coronary Angiography, Coronary Artery Disease, Radiology Information Systems. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 41 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: National Research Foundation of Korea, Seoul National University Hospital.
Abstract (Originalsprache)
OBJECTIVE: To evaluate the accuracy of large language models (LLMs) in extracting Coronary Artery Disease-Reporting and Data System (CAD-RADS) 2.0 components from coronary CT angiography (CCTA) reports, and assess the impact of prompting strategies. MATERIALS AND METHODS: In this multi-institutional study, we collected 319 synthetic, semi-structured CCTA reports from six institutions to protect patient privacy while maintaining clinical relevance. The dataset included 150 reports from a primary institution (100 for instruction development and 50 for internal testing) and 169 reports from five external institutions for external testing. Board-certified radiologists established reference standards following the CAD-RADS 2.0 guidelines for all three components: stenosis severity, plaque burden, and modifiers. Six LLMs (GPT-4, GPT-4o, Claude-3.5-Sonnet, o1-mini, Gemini-1.5-Pro, and DeepSeek-R1-Distill-Qwen-14B) were evaluated using an optimized instruction with prompting strategies, including zero-shot or few-shot with or without chain-of-thought (CoT) prompting. The accuracy was assessed and compared using McNemar's test. RESULTS: LLMs demonstrated robust accuracy across all CAD-RADS 2.0 components. Peak stenosis severity accuracies reached 0.980 (48/49, Claude-3.5-Sonnet and o1-mini) in internal testing and 0.946 (158/167, GPT-4o and o1-mini) in external testing. Plaque burden extraction showed exceptional accuracy, with multiple models achieving perfect accuracy (43/43) in internal testing and 0.993 (137/138, GPT-4o, and o1-mini) in external testing. Modifier detection demonstrated consistently high accuracy (≥0.990) across most models. One open-source model, DeepSeek-R1-Distill-Qwen-14B, showed a relatively low accuracy for stenosis severity: 0.898 (44/49, internal) and 0.820 (137/167, external). CoT prompting significantly enhanced the accuracy of several models, with GPT-4 showing the most substantial improvements: stenosis severity accuracy increased by 0.192 (P < 0.001) and plaque burden accuracy by 0.152 (P < 0.001) in external testing. CONCLUSION: LLMs demonstrated high accuracy in automated extraction of CAD-RADS 2.0 components from semi-structured CCTA reports, particularly when used with CoT prompting.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Interdisciplinary Program in Bioengineering, Seoul National University Graduate School, Seoul, Republic of Korea.
- Integrated Major in Innovative Medical Science, Seoul National University Graduate School, Seoul, Republic of Korea.
- Department of Radiology, Boramae Medical Center, Seoul National University College of Medicine, Seoul, Republic of Korea.
- Seoul National University College of Medicine, Seoul, Republic of Korea.
- Division of Cardiology, Department of Internal Medicine, Boramae Medical Center, Seoul National University College of Medicine, Seoul, Republic of Korea.
- Department of Radiology, Chonnam National University Hwasun Hospital and Chonnam National University Medical School, Hwasun, Republic of Korea.
ORCID-Profile: Dabin Min, Kwang Nam Jin, SangHeum Bang, Moon Young Kim, Hack-Lyoung Kim, Won Gi Jeong, Hye-Jeong Lee, Kyongmin Sarah Beck, Sung Ho Hwang, Eun Young Kim, Chang Min Park
Förderung
- National Research Foundation of Korea
- Seoul National University Hospital
Zitieren
APA
Min, D., Jin, K. N., Bang, S., Kim, M. Y., Kim, H. L., Jeong, W. G., Lee, H. J., Beck, K. S., Hwang, S. H., Kim, E. Y., & Park, C. M. (2025). Large Language Models for CAD-RADS 2.0 Extraction From Semi-Structured Coronary CT Angiography Reports: A Multi-Institutional Study. Korean Journal of Radiology, 26(9), 817. https://doi.org/10.3348/kjr.2025.0293
Vancouver
Min D, Jin KN, Bang S, Kim MY, Kim HL, Jeong WG, et al. Large Language Models for CAD-RADS 2.0 Extraction From Semi-Structured Coronary CT Angiography Reports: A Multi-Institutional Study. Korean Journal of Radiology. 2025;26(9):817. doi:10.3348/kjr.2025.0293
BibTeX
@article{Min2025,
author = {Min, Dabin and Jin, Kwang Nam and Bang, SangHeum and Kim, Moon Young and Kim, Hack-Lyoung and Jeong, Won Gi and Lee, Hye-Jeong and Beck, Kyongmin Sarah and Hwang, Sung Ho and Kim, Eun Young and Park, Chang Min},
title = {Large Language Models for CAD-RADS 2.0 Extraction From Semi-Structured Coronary CT Angiography Reports: A Multi-Institutional Study},
journal = {Korean Journal of Radiology},
year = {2025},
volume = {26},
number = {9},
pages = {817},
doi = {10.3348/kjr.2025.0293}
}
EUVIMED