Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework
JMIR Medical Informatics · 2025
DOI: 10.2196/68618 ↗At a glance
Journal article in JMIR Medical Informatics, 2025. Written by Abdullah Abdullah and 1 co-authors. Institution: Department of Computer Science and Engineering, Kyung Hee University, 1732 Deogyeong-daero, Giheung-gu, Yongin, Gyeonggi-do, 17104, Republic of Korea…. Major MEDLINE topics: Radiography, Thoracic, Natural Language Processing, Radiology Information Systems. Cited 15 times so far (Crossref); 35 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Labeling unstructured radiology reports is crucial for creating structured datasets that facilitate downstream tasks, such as training large-scale medical imaging models. Current approaches typically rely on Bidirectional Encoder Representations from Transformers (BERT)-based methods or manual expert annotations, which have limitations in terms of scalability and performance.
Objective
This study aimed to evaluate the effectiveness of a generative pretrained transformer (GPT)-based large language model (LLM) in labeling radiology reports, comparing it with 2 existing methods, CheXbert and CheXpert, on a large chest X-ray dataset (MIMIC Chest X-ray [MIMIC-CXR]).
Methods
In this study, we introduce an LLM-based approach fine-tuned on expert-labeled radiology reports. Our model’s performance was evaluated on 687 radiologist-labeled chest X-ray reports, comparing F 1 scores across 14 thoracic pathologies. The performance of our LLM model was compared with the CheXbert and CheXpert models across positive, negative, and uncertainty extraction tasks. Paired t tests and Wilcoxon signed-rank tests were performed to evaluate the statistical significance of differences between model performances.
Results
The GPT-based LLM model achieved an average F 1 score of 0.9014 across all certainty levels, outperforming CheXpert (0.8864) and approaching CheXbert’s performance (0.9047). For positive and negative certainty levels, our model scored 0.8708, surpassing CheXpert (0.8525) and closely matching CheXbert (0.8733). Statistically, paired t tests indicated no significant difference between our model and CheXbert ( P =.35) but a significant improvement over CheXpert ( P =.). Wilcoxon signed-rank tests corroborated these findings, showing no significant difference between our model and CheXbert ( P =.4) but confirming a significant difference with CheXpert ( P =.). The LLM also demonstrated superior performance for pathologies with longer and more complex descriptions, leveraging its extended context length.
Conclusions
The GPT-based LLM model demonstrates competitive performance compared with CheXbert and outperforms CheXpert in radiology report labeling. These findings suggest that LLMs are a promising alternative to traditional BERT-based architectures for this task, offering enhanced context understanding and eliminating the need for extensive feature engineering. Furthermore, with large context length LLM-based models are better suited for this task as compared with the small context length of BERT based models.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Computer Science and Engineering, Kyung Hee University, 1732 Deogyeong-daero, Giheung-gu, Yongin, Gyeonggi-do, 17104, Republic of Korea, 82 312013761.
ORCID profiles: Abdullah Abdullah, Seong Tae Kim
Cite
APA
Abdullah, A., & Kim, S. T. (2025). Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework. JMIR Medical Informatics. https://doi.org/10.2196/68618
Vancouver
Abdullah A, Kim ST. Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework. JMIR Medical Informatics. 2025. doi:10.2196/68618
BibTeX
@article{Abdullah2025,
author = {Abdullah, Abdullah and Kim, Seong Tae},
title = {Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework},
journal = {JMIR Medical Informatics},
year = {2025},
doi = {10.2196/68618}
}
EUVIMED