EUVIMEDEUVIMED entdecken
Open Access

Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework

Abdullah Abdullah iD, Seong Tae Kim iD

JMIR Medical Informatics · 2025

DOI: 10.2196/68618 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Informatics, 2025. Verfasst von Abdullah Abdullah und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science and Engineering, Kyung Hee University, 1732 Deogyeong-daero, Giheung-gu, Yongin, Gyeonggi-do, 17104, Republic of Korea…. Hauptthemen laut MEDLINE: Radiography, Thoracic, Natural Language Processing, Radiology Information Systems. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Labeling unstructured radiology reports is crucial for creating structured datasets that facilitate downstream tasks, such as training large-scale medical imaging models. Current approaches typically rely on Bidirectional Encoder Representations from Transformers (BERT)-based methods or manual expert annotations, which have limitations in terms of scalability and performance.

Objective

This study aimed to evaluate the effectiveness of a generative pretrained transformer (GPT)-based large language model (LLM) in labeling radiology reports, comparing it with 2 existing methods, CheXbert and CheXpert, on a large chest X-ray dataset (MIMIC Chest X-ray [MIMIC-CXR]).

Methods

In this study, we introduce an LLM-based approach fine-tuned on expert-labeled radiology reports. Our model’s performance was evaluated on 687 radiologist-labeled chest X-ray reports, comparing F 1 scores across 14 thoracic pathologies. The performance of our LLM model was compared with the CheXbert and CheXpert models across positive, negative, and uncertainty extraction tasks. Paired t tests and Wilcoxon signed-rank tests were performed to evaluate the statistical significance of differences between model performances.

Results

The GPT-based LLM model achieved an average F 1 score of 0.9014 across all certainty levels, outperforming CheXpert (0.8864) and approaching CheXbert’s performance (0.9047). For positive and negative certainty levels, our model scored 0.8708, surpassing CheXpert (0.8525) and closely matching CheXbert (0.8733). Statistically, paired t tests indicated no significant difference between our model and CheXbert ( P =.35) but a significant improvement over CheXpert ( P =.). Wilcoxon signed-rank tests corroborated these findings, showing no significant difference between our model and CheXbert ( P =.4) but confirming a significant difference with CheXpert ( P =.). The LLM also demonstrated superior performance for pathologies with longer and more complex descriptions, leveraging its extended context length.

Conclusions

The GPT-based LLM model demonstrates competitive performance compared with CheXbert and outperforms CheXpert in radiology report labeling. These findings suggest that LLMs are a promising alternative to traditional BERT-based architectures for this task, offering enhanced context understanding and eliminating the need for extensive feature engineering. Furthermore, with large context length LLM-based models are better suited for this task as compared with the small context length of BERT based models.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Radiography, ThoracicNatural Language ProcessingRadiology Information SystemsHumansLarge Language Models

Stichwörter der Autor:innen

BERTGPTLLMgenerative pre-trained transformerslabelinglarge language modelradiology reportthoracic pathologies

Autor:innen und Institutionen

  • Department of Computer Science and Engineering, Kyung Hee University, 1732 Deogyeong-daero, Giheung-gu, Yongin, Gyeonggi-do, 17104, Republic of Korea, 82 312013761.

ORCID-Profile: Abdullah Abdullah, Seong Tae Kim

Zitieren

APA

Abdullah, A., & Kim, S. T. (2025). Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework. JMIR Medical Informatics. https://doi.org/10.2196/68618

Vancouver

Abdullah A, Kim ST. Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework. JMIR Medical Informatics. 2025. doi:10.2196/68618

BibTeX
@article{Abdullah2025,
  author = {Abdullah, Abdullah and Kim, Seong Tae},
  title = {Automated Radiology Report Labeling in Chest X-Ray Pathologies: Development and Evaluation of a Large Language Model Framework},
  journal = {JMIR Medical Informatics},
  year = {2025},
  doi = {10.2196/68618}
}
Kostenlos starten →