Assessing Large Language Models for Oncology Data Inference From Radiology Reports
JCO Clinical Cancer Informatics · 2024-12 · (8)
DOI: 10.1200/cci.24.00126 ↗Kurz gefasst
Fachartikel in der Zeitschrift JCO Clinical Cancer Informatics, 2024-12. Verfasst von Li-Ching Chen und 8 weiteren Autor:innen. Beteiligte Einrichtung: University of California, Berkeley, Berkeley, CA. Hauptthemen laut MEDLINE: Natural Language Processing. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 43 Literaturangaben. Gefördert von: NCI NIH HHS.
Abstract (Originalsprache)
PURPOSE We examined the effectiveness of proprietary and open large language models (LLMs) in detecting disease presence, location, and treatment response in pancreatic cancer from radiology reports. METHODS We analyzed 203 deidentified radiology reports, manually annotated for disease status, location, and indeterminate nodules needing follow-up. Using generative pre-trained transformer (GPT)-4, GPT-3.5-turbo, and open models such as Gemma-7B and Llama3-8B, we employed strategies such as ablation and prompt engineering to boost accuracy. Discrepancies between human and model interpretations were reviewed by a secondary oncologist. RESULTS Among 164 patients with pancreatic tumor, GPT-4 showed the highest accuracy in inferring disease status, achieving a 75.5% correctness (F1-micro). Open models Mistral-7B and Llama3-8B performed comparably, with accuracies of 68.6% and 61.4%, respectively. Mistral-7B excelled in deriving correct inferences from objective findings directly. Most tested models demonstrated proficiency in identifying disease containing anatomic locations from a list of choices, with GPT-4 and Llama3-8B showing near-parity in precision and recall for disease site identification. However, open models struggled with differentiating benign from malignant postsurgical changes, affecting their precision in identifying findings indeterminate for cancer. A secondary review occasionally favored GPT-3.5's interpretations, indicating the variability in human judgment. CONCLUSION LLMs, especially GPT-4, are proficient in deriving oncologic insights from radiology reports. Their performance is enhanced by effective summarization strategies, demonstrating their potential in clinical support and health care analytics. This study also underscores the possibility of zero-shot open model utility in environments where proprietary models are restricted. Finally, by providing a set of annotated radiology reports, this paper presents a valuable data set for further LLM research in oncology.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- University of California, Berkeley, Berkeley, CA.
- University of California, San Francisco, San Francisco, CA.
- Bakar Computational Health Sciences Institute, University of California, San Francisco, San Francisco, CA.
- Helen Diller Family Comprehensive Cancer Center, University of California, San Francisco, San Francisco, CA.
ORCID-Profile: Li-Ching Chen, Travis Zack, Madhumita Sushil, Brenda Miao, Corynn Kasap, Atul Butte, Eric A Collisson, Julian C Hong
Förderung
- NCI NIH HHS
Zitieren
APA
Chen, L. C., Zack, T., Demirci, A., Sushil, M., Miao, B., Kasap, C., Butte, A., Collisson, E. A., & Hong, J. C. (2024). Assessing Large Language Models for Oncology Data Inference From Radiology Reports. JCO Clinical Cancer Informatics. https://doi.org/10.1200/cci.24.00126
Vancouver
Chen LC, Zack T, Demirci A, Sushil M, Miao B, Kasap C, et al. Assessing Large Language Models for Oncology Data Inference From Radiology Reports. JCO Clinical Cancer Informatics. 2024. doi:10.1200/cci.24.00126
BibTeX
@article{Chen2024,
author = {Chen, Li-Ching and Zack, Travis and Demirci, Arda and Sushil, Madhumita and Miao, Brenda and Kasap, Corynn and Butte, Atul and Collisson, Eric A and Hong, Julian C},
title = {Assessing Large Language Models for Oncology Data Inference From Radiology Reports},
journal = {JCO Clinical Cancer Informatics},
year = {2024},
number = {8},
doi = {10.1200/cci.24.00126}
}
EUVIMED