Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework
JMIR Medical Informatics · 2025
DOI: 10.2196/68618 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Medical Informatics, 2025. Verfasst von Abdullah Abdullah und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Engineering, Kyung Hee University, 1732 Deogyeong-daero, Giheung-gu, Yongin, Gyeonggi-do, 17104, Republic of Korea…. Hauptthemen laut MEDLINE: Radiography, Thoracic, Natural Language Processing, Radiology Information Systems. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Labeling unstructured radiology reports is crucial for creating structured datasets that facilitate downstream tasks, such as training large-scale medical imaging models. Current approaches typically rely on Bidirectional Encoder Representations from Transformers (BERT)-based methods or manual expert annotations, which have limitations in terms of scalability and performance.
Objective
This study aimed to evaluate the effectiveness of a generative pretrained transformer (GPT)-based large language model (LLM) in labeling radiology reports, comparing it with 2 existing methods, CheXbert and CheXpert, on a large chest X-ray dataset (MIMIC Chest X-ray [MIMIC-CXR]).
Methods
In this study, we introduce an LLM-based approach fine-tuned on expert-labeled radiology reports. Our model’s performance was evaluated on 687 radiologist-labeled chest X-ray reports, comparing F 1 scores across 14 thoracic pathologies. The performance of our LLM model was compared with the CheXbert and CheXpert models across positive, negative, and uncertainty extraction tasks. Paired t tests and Wilcoxon signed-rank tests were performed to evaluate the statistical significance of differences between model performances.
Results
The GPT-based LLM model achieved an average F 1 score of 0.9014 across all certainty levels, outperforming CheXpert (0.8864) and approaching CheXbert’s performance (0.9047). For positive and negative certainty levels, our model scored 0.8708, surpassing CheXpert (0.8525) and closely matching CheXbert (0.8733). Statistically, paired t tests indicated no significant difference between our model and CheXbert ( P =.35) but a significant improvement over CheXpert ( P =.). Wilcoxon signed-rank tests corroborated these findings, showing no significant difference between our model and CheXbert ( P =.4) but confirming a significant difference with CheXpert ( P =.). The LLM also demonstrated superior performance for pathologies with longer and more complex descriptions, leveraging its extended context length.
Conclusions
The GPT-based LLM model demonstrates competitive performance compared with CheXbert and outperforms CheXpert in radiology report labeling. These findings suggest that LLMs are a promising alternative to traditional BERT-based architectures for this task, offering enhanced context understanding and eliminating the need for extensive feature engineering. Furthermore, with large context length LLM-based models are better suited for this task as compared with the small context length of BERT based models.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science and Engineering, Kyung Hee University, 1732 Deogyeong-daero, Giheung-gu, Yongin, Gyeonggi-do, 17104, Republic of Korea, 82 312013761.
ORCID-Profile: Abdullah Abdullah, Seong Tae Kim
Zitieren
APA
Abdullah, A., & Kim, S. T. (2025). Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework. JMIR Medical Informatics. https://doi.org/10.2196/68618
Vancouver
Abdullah A, Kim ST. Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework. JMIR Medical Informatics. 2025. doi:10.2196/68618
BibTeX
@article{Abdullah2025,
author = {Abdullah, Abdullah and Kim, Seong Tae},
title = {Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework},
journal = {JMIR Medical Informatics},
year = {2025},
doi = {10.2196/68618}
}
EUVIMED