EUVIMEDEUVIMED entdecken
VergleichsstudieOpen Access · CC BY-NC-ND 4.0

A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports

Madhumita Sushil iD, Travis Zack, Divneet Mandair, Zhiwei Zheng, Ahmed Wali, Yan-Ning Yu, Yuwei Quan, Dmytro Lituiev iD, Atul J Butte

Journal of the American Medical Informatics Association · 01.10.2024 · 31 (10) · 2315-2327

DOI: 10.1093/jamia/ocae146 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 20.06.2024. Verfasst von Madhumita Sushil und 8 weiteren Autor:innen. Beteiligte Einrichtung: Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, CA 94158, United States. Hauptthemen laut MEDLINE: Breast Neoplasms, Supervised Machine Learning. Die Arbeit wurde bisher 52-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: NCATS NIH HHS, UCSF-Stanford Center of Excellence in Regulatory Sciences and Innovation, FDA HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

Although supervised machine learning is popular for information extraction from clinical notes, creating large annotated datasets requires extensive domain expertise and is time-consuming. Meanwhile, large language models (LLMs) have demonstrated promising transfer learning capability. In this study, we explored whether recent LLMs could reduce the need for large-scale data annotations. Materials and Methods We curated a dataset of 769 breast cancer pathology reports, manually labeled with 12 categories, to compare zero-shot classification capability of the following LLMs: GPT-4, GPT-3.5, Starling, and ClinicalCamel, with task-specific supervised classification performance of 3 models: random forests, long short-term memory networks with attention (LSTM-Att), and the UCSF-BERT model.

Results

Across all 12 tasks, the GPT-4 model performed either significantly better than or as well as the best supervised model, LSTM-Att (average macro F1-score of 0.86 vs 0.75), with advantage on tasks with high label imbalance. Other LLMs demonstrated poor performance. Frequent GPT-4 error categories included incorrect inferences from multiple samples and from history, and complex task design, and several LSTM-Att errors were related to poor generalization to the test set.

Discussion

On tasks where large annotated datasets cannot be easily collected, LLMs can reduce the burden of data labeling. However, if the use of LLMs is prohibitive, the use of simpler models with large annotated datasets can provide comparable results.

Conclusions

GPT-4 demonstrated the potential to speed up the execution of clinical NLP studies by reducing the need for large annotated datasets. This may increase the utilization of NLP-based variables and outcomes in clinical studies.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Breast NeoplasmsSupervised Machine LearningHumansFemaleNatural Language ProcessingDatasets as TopicElectronic Health RecordsData Mining

Stichwörter der Autor:innen

breast cancerelectronic health recordslarge language modelsnatural language processingpathology

Autor:innen und Institutionen

  • Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, CA 94158, United States.
  • Helen Diller Family Comprehensive Cancer Center, University of California, San Francisco, San Francisco, CA 94158, United States.
  • University of California, Berkeley, Berkeley, CA 94720, United States.
  • Center for Data-driven Insights and Innovation, University of California, Office of the President, Oakland, CA 94607, United States.
  • Department of Pediatrics, University of California, San Francisco, San Francisco, CA 94158, United States.

ORCID-Profile: Madhumita Sushil, Dmytro Lituiev

Förderung

  • NCATS NIH HHS
  • UCSF-Stanford Center of Excellence in Regulatory Sciences and Innovation
  • FDA HHS
  • NIH HHS
  • NCI NIH HHS
  • National Cancer Institute
  • National Institutes of Health
  • FDA

Zitieren

APA

Sushil, M., Zack, T., Mandair, D., Zheng, Z., Wali, A., Yu, Y. N., Quan, Y., Lituiev, D., & Butte, A. J. (2024). A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports. Journal of the American Medical Informatics Association, 31(10), 2315-2327. https://doi.org/10.1093/jamia/ocae146

Vancouver

Sushil M, Zack T, Mandair D, Zheng Z, Wali A, Yu YN, et al. A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports. Journal of the American Medical Informatics Association. 2024;31(10):2315-2327. doi:10.1093/jamia/ocae146

BibTeX
@article{Sushil2024,
  author = {Sushil, Madhumita and Zack, Travis and Mandair, Divneet and Zheng, Zhiwei and Wali, Ahmed and Yu, Yan-Ning and Quan, Yuwei and Lituiev, Dmytro and Butte, Atul J},
  title = {A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports},
  journal = {Journal of the American Medical Informatics Association},
  year = {2024},
  volume = {31},
  number = {10},
  pages = {2315--2327},
  doi = {10.1093/jamia/ocae146}
}
Kostenlos starten →