EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Limitations of public chest radiography datasets for artificial intelligence: label quality, domain shift, bias and evaluation challenges

Amy Rafferty iD, Ajitha Rajan iD

Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · 16.07.2026 · 384 (2324)

DOI: 10.1098/rsta.2025.0129 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, online erschienen am 16.07.2026. Verfasst von Amy Rafferty und 1 weiteren Autor:innen. Beteiligte Einrichtung: School of Informatics, The University of Edinburgh , Edinburgh, UK. Hauptthemen laut MEDLINE: Radiography, Thoracic, Artificial Intelligence, Databases, Factual. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Artificial intelligence (AI) has shown significant promise in chest radiography, where deep learning models can approach radiologist-level diagnostic performance. Progress has been accelerated by large public datasets, such as MIMIC-CXR, ChestX-ray14, PadChest and CheXpert, which provide hundreds of thousands of labelled images with pathology annotations. However, these datasets also present important limitations. Automated label extraction from radiology reports introduces errors, particularly in handling uncertainty and negation, and radiologist review frequently disagrees with assigned labels. In addition, domain shift and population bias restrict model generalizability, while evaluation practices often overlook clinically meaningful measures. We conduct a systematic analysis of these challenges, focusing on label quality, dataset bias and domain shift. Our cross-dataset domain-shift evaluation across multiple model architectures revealed substantial external performance degradation, with pronounced reductions in area under the precision–recall curve (AUPRC) and F1 scores compared with internal testing. To assess dataset bias, we trained a source-classification model that distinguished datasets with near-perfect accuracy, and performed subgroup analyses showing reduced performance for minority age and sex groups. Finally, expert review by two board-certified radiologists identified significant disagreement with public dataset labels. Our findings highlight important clinical weaknesses in current benchmarks and emphasize the need for clinician-validated datasets and fairer evaluation frameworks. This article is part of the theme issue ‘Safe, secure and robust AI for safety-critical systems’.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Radiography, ThoracicArtificial IntelligenceDatabases, FactualHumansDeep LearningBias

Stichwörter der Autor:innen

chest radiographydataset qualitydeep learningmachine learning

Autor:innen und Institutionen

  • School of Informatics, The University of Edinburgh , Edinburgh, UK.

ORCID-Profile: Amy Rafferty, Ajitha Rajan

Zitieren

APA

Rafferty, A., & Rajan, A. (2026). Limitations of public chest radiography datasets for artificial intelligence: label quality, domain shift, bias and evaluation challenges. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 384(2324). https://doi.org/10.1098/rsta.2025.0129

Vancouver

Rafferty A, Rajan A. Limitations of public chest radiography datasets for artificial intelligence: label quality, domain shift, bias and evaluation challenges. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 2026;384(2324). doi:10.1098/rsta.2025.0129

BibTeX
@article{Rafferty2026,
  author = {Rafferty, Amy and Rajan, Ajitha},
  title = {Limitations of public chest radiography datasets for artificial intelligence: label quality, domain shift, bias and evaluation challenges},
  journal = {Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences},
  year = {2026},
  volume = {384},
  number = {2324},
  doi = {10.1098/rsta.2025.0129}
}
Kostenlos starten →