EUVIMEDEUVIMED entdecken
Open Access

Transformer versus traditional natural language processing: how much data is enough for automated radiology report classification?

Eric Yang, Matthew D Li, Shruti Raghavan, Francis Deng, Min Lang, Marc D Succi, Ambrose J Huang, Jayashree Kalpathy-Cramer

The British Journal of Radiology · 2023

DOI: 10.1259/bjr.20220769 ↗

Kurz gefasst

Fachartikel in der Zeitschrift The British Journal of Radiology, 2023. Verfasst von Eric Yang und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA. Hauptthemen laut MEDLINE: Natural Language Processing, Radiology. Die Arbeit wurde bisher 35-mal zitiert (Crossref) und nennt 24 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIBIB NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Current state-of-the-art natural language processing (NLP) techniques use transformer deep-learning architectures, which depend on large training datasets. We hypothesized that traditional NLP techniques may outperform transformers for smaller radiology report datasets.

Methods

We compared the performance of BioBERT, a deep-learning-based transformer model pre-trained on biomedical text, and three traditional machine-learning models (gradient boosted tree, random forest, and logistic regression) on seven classification tasks given free-text radiology reports. Tasks included detection of appendicitis, diverticulitis, bowel obstruction, and enteritis/colitis on abdomen/pelvis CT reports, ischemic infarct on brain CT/MRI reports, and medial and lateral meniscus tears on knee MRI reports (7,204 total annotated reports). The performance of NLP models on held-out test sets was compared after training using the full training set, and 2.5%, 10%, 25%, 50%, and 75% random subsets of the training data.

Results

In all tested classification tasks, BioBERT performed poorly at smaller training sample sizes compared to non-deep-learning NLP models. Specifically, BioBERT required training on approximately 1,000 reports to perform similarly or better than non-deep-learning models. At around 1,250 to 1,500 training samples, the testing performance for all models began to plateau, where additional training data yielded minimal performance gain.

Conclusions

With larger sample sizes, transformer NLP models achieved superior performance in radiology report binary classification tasks. However, with smaller sizes (<1000) and more imbalanced training data, traditional NLP techniques performed better. Advances in knowledge: Our benchmarks can help guide clinical NLP researchers in selecting machine-learning models according to their dataset characteristics.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingRadiologyHumansTomography, X-Ray ComputedMachine LearningMagnetic Resonance Imaging

Autor:innen und Institutionen

  • Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA.
  • Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
  • Department of Radiology and Diagnostic Imaging, University of Alberta, Edmonton, Alberta, Canada.
  • Department of Ophthalmology, University of Colorado, Aurora, CO, USA.

Förderung

  • NIBIB NIH HHS

Zitieren

APA

Yang, E., Li, M. D., Raghavan, S., Deng, F., Lang, M., Succi, M. D., Huang, A. J., & Kalpathy-Cramer, J. (2023). Transformer versus traditional natural language processing: how much data is enough for automated radiology report classification?. The British Journal of Radiology. https://doi.org/10.1259/bjr.20220769

Vancouver

Yang E, Li MD, Raghavan S, Deng F, Lang M, Succi MD, et al. Transformer versus traditional natural language processing: how much data is enough for automated radiology report classification?. The British Journal of Radiology. 2023. doi:10.1259/bjr.20220769

BibTeX
@article{Yang2023,
  author = {Yang, Eric and Li, Matthew D and Raghavan, Shruti and Deng, Francis and Lang, Min and Succi, Marc D and Huang, Ambrose J and Kalpathy-Cramer, Jayashree},
  title = {Transformer versus traditional natural language processing: how much data is enough for automated radiology report classification?},
  journal = {The British Journal of Radiology},
  year = {2023},
  doi = {10.1259/bjr.20220769}
}
Kostenlos starten →