A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports
Journal of the American Medical Informatics Association · 01.10.2024 · 31 (10) · 2315-2327
DOI: 10.1093/jamia/ocae146 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 20.06.2024. Verfasst von Madhumita Sushil und 8 weiteren Autor:innen. Beteiligte Einrichtung: Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, CA 94158, United States. Hauptthemen laut MEDLINE: Breast Neoplasms, Supervised Machine Learning. Die Arbeit wurde bisher 52-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0). Gefördert von: NCATS NIH HHS, UCSF-Stanford Center of Excellence in Regulatory Sciences and Innovation, FDA HHS.
Abstract (Originalsprache)
Objective
Although supervised machine learning is popular for information extraction from clinical notes, creating large annotated datasets requires extensive domain expertise and is time-consuming. Meanwhile, large language models (LLMs) have demonstrated promising transfer learning capability. In this study, we explored whether recent LLMs could reduce the need for large-scale data annotations. Materials and Methods We curated a dataset of 769 breast cancer pathology reports, manually labeled with 12 categories, to compare zero-shot classification capability of the following LLMs: GPT-4, GPT-3.5, Starling, and ClinicalCamel, with task-specific supervised classification performance of 3 models: random forests, long short-term memory networks with attention (LSTM-Att), and the UCSF-BERT model.
Results
Across all 12 tasks, the GPT-4 model performed either significantly better than or as well as the best supervised model, LSTM-Att (average macro F1-score of 0.86 vs 0.75), with advantage on tasks with high label imbalance. Other LLMs demonstrated poor performance. Frequent GPT-4 error categories included incorrect inferences from multiple samples and from history, and complex task design, and several LSTM-Att errors were related to poor generalization to the test set.
Discussion
On tasks where large annotated datasets cannot be easily collected, LLMs can reduce the burden of data labeling. However, if the use of LLMs is prohibitive, the use of simpler models with large annotated datasets can provide comparable results.
Conclusions
GPT-4 demonstrated the potential to speed up the execution of clinical NLP studies by reducing the need for large annotated datasets. This may increase the utilization of NLP-based variables and outcomes in clinical studies.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, CA 94158, United States.
- Helen Diller Family Comprehensive Cancer Center, University of California, San Francisco, San Francisco, CA 94158, United States.
- University of California, Berkeley, Berkeley, CA 94720, United States.
- Center for Data-driven Insights and Innovation, University of California, Office of the President, Oakland, CA 94607, United States.
- Department of Pediatrics, University of California, San Francisco, San Francisco, CA 94158, United States.
ORCID-Profile: Madhumita Sushil, Dmytro Lituiev
Förderung
- NCATS NIH HHS
- UCSF-Stanford Center of Excellence in Regulatory Sciences and Innovation
- FDA HHS
- NIH HHS
- NCI NIH HHS
- National Cancer Institute
- National Institutes of Health
- FDA
Zitieren
APA
Sushil, M., Zack, T., Mandair, D., Zheng, Z., Wali, A., Yu, Y. N., Quan, Y., Lituiev, D., & Butte, A. J. (2024). A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports. Journal of the American Medical Informatics Association, 31(10), 2315-2327. https://doi.org/10.1093/jamia/ocae146
Vancouver
Sushil M, Zack T, Mandair D, Zheng Z, Wali A, Yu YN, et al. A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports. Journal of the American Medical Informatics Association. 2024;31(10):2315-2327. doi:10.1093/jamia/ocae146
BibTeX
@article{Sushil2024,
author = {Sushil, Madhumita and Zack, Travis and Mandair, Divneet and Zheng, Zhiwei and Wali, Ahmed and Yu, Yan-Ning and Quan, Yuwei and Lituiev, Dmytro and Butte, Atul J},
title = {A comparative study of large language model-based zero-shot inference and task-specific supervised classification of breast cancer pathology reports},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
volume = {31},
number = {10},
pages = {2315--2327},
doi = {10.1093/jamia/ocae146}
}
EUVIMED