EUVIMEDSkopri EUVIMED
Multizentrische StudieOpen access

Large Language Model–Assisted Radiology Report Quality Assurance: Real-World Multicenter Evaluation

Lujing Wang, Yushuang Zhang, Liang Peng, Pei Liu, Li Wang, Yu Cui, Yixin Chen, Xingyu Li, Yitian Tao, Xiuxian Gong, Xiaoying Zhao, Yujiao Song and 5 more

Journal of Medical Internet Research · 2026-10-28 · 28 · e106250-e106250

DOI: 10.2196/106250 ↗

At a glance

Multizentrische Studie in Journal of Medical Internet Research, published online on 2026-10-28. Written by Lujing Wang and 16 co-authors. Institution: Department of Radiology, The Second Affiliated Hospital of Kunming Medical University, Kunming, Yunnan, 650101, China, 86 0871-63402918. Major MEDLINE topics: Large Language Models, Quality Assurance, Health Care, Radiology. 32 references. The full text is freely available in PubMed Central.

AstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Background

Radiology reports serve as the definitive clinical record of imaging diagnoses, with their accuracy being directly critical to patient safety. However, amid escalating workloads, manual review remains inadequate for capturing subtle errors in reports on a large scale. Although large language models (LLMs) have demonstrated considerable promise in medical text processing, previous studies have predominantly been confined to radiology report text from a single imaging modality or datasets with synthetic errors.

Objective

This study aimed to evaluate the feasibility, accuracy, and generalizability of general-purpose LLMs for quality assurance of radiology report text from multiple imaging modalities (x-ray, computed tomography [CT], and magnetic resonance imaging [MRI]), multi-anatomical regions, and multicenter clinical settings, with head-to-head comparisons conducted against radiologists of varying seniority.

Methods

Radiology report texts from 3 aforementioned imaging modalities and 5 anatomical regions (head, chest, abdomen, spine, and knee) were retrospectively collected from center 1 (March 2025-May 2025) and reviewed using a clinician-annotated 5-category error taxonomy (type 1: interpretive; type 2: factual). Five LLMs (Qwen3, GPT-5.4 Pro, Gemini 3.1 Pro, Llama 4, and DeepSeek-V3.1) were benchmarked on 10,000 randomly selected reports, and the best-performing model was evaluated on the full cohort. A blinded head-to-head comparison with 3 radiologists (3-10 years of experience) used a modality-stratified, error-enriched challenge set. The radiologists subsequently rated the model-generated explanations using a 5-point Likert scale. Cross-center performance was evaluated in an independent external cohort. The same 2 senior radiologists labeled reports from both centers using the same protocol, resolving disagreements by consensus.

Results

In total, 4.6% (4431/95,582) of the reports in center 1 contained at least one confirmed text internal error. Qwen3, the selected model, achieved a report-level sensitivity, specificity, positive predictive value, and F 1 -score of 0.99, 0.99, 0.90, and 0.94, respectively, with high performance across reports from the 3 imaging modalities and multiple anatomical regions and better detection of factual than interpretive errors. In the error-enriched challenge set, its error instance exact category recall exceeded that of the radiologists (0.96 vs 0.45-0.67; P

Subjects

MeSH terms (MEDLINE)

Large Language ModelsQuality Assurance, Health CareRadiologyHumansRetrospective StudiesTomography, X-Ray ComputedMagnetic Resonance Imaging

Author keywords

AIartificial intelligenceclinical AIexternal validationhuman evaluationlarge language modelquality assuranceradiology reportreport error

Authors and institutions

  • Department of Radiology, The Second Affiliated Hospital of Kunming Medical University, Kunming, Yunnan, 650101, China, 86 0871-63402918.
  • Department of Radiology, The Sixth Affiliated Hospital of Kunming Medical University, Yuxi, Yunnan, China.
  • School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, China.
  • School of Biomedical Engineering, ShanghaiTech University, Shanghai, China.
  • Department of Radiology, School of Clinical Medicine, University of Cambridge, Cambridge, United Kingdom.
  • Department of Radiology, National Center for Cardiovascular Diseases, Fuwai Hospital, Beijing, China.

Cite

APA

Wang, L., Zhang, Y., Peng, L., Liu, P., Wang, L., Cui, Y., Chen, Y., Li, X., Tao, Y., Gong, X., Zhao, X., Song, Y., Chen, J., Zhao, S., Wang, F., Ma, J., & Zhao, X. (2026). Large Language Model–Assisted Radiology Report Quality Assurance: Real-World Multicenter Evaluation. Journal of Medical Internet Research, 28, e106250-e106250. https://doi.org/10.2196/106250

Vancouver

Wang L, Zhang Y, Peng L, Liu P, Wang L, Cui Y, et al. Large Language Model–Assisted Radiology Report Quality Assurance: Real-World Multicenter Evaluation. Journal of Medical Internet Research. 2026;28:e106250-e106250. doi:10.2196/106250

BibTeX
@article{Wang2026,
  author = {Wang, Lujing and Zhang, Yushuang and Peng, Liang and Liu, Pei and Wang, Li and Cui, Yu and Chen, Yixin and Li, Xingyu and Tao, Yitian and Gong, Xiuxian and Zhao, Xiaoying and Song, Yujiao and Chen, Jian and Zhao, Shihua and Wang, Fuying and Ma, Jian and Zhao, Xinxiang},
  title = {Large Language Model–Assisted Radiology Report Quality Assurance: Real-World Multicenter Evaluation},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e106250--e106250},
  doi = {10.2196/106250}
}
Start for free →