Inferring cancer disease response from radiology reports using large language models with data augmentation and prompting
Journal of the American Medical Informatics Association · 25.09.2023 · 30 (10) · 1657-1664
DOI: 10.1093/jamia/ocad133 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 14.07.2023. Verfasst von Ryan Shea Ying Cong Tan und 23 weiteren Autor:innen. Beteiligte Einrichtung: Division of Medical Oncology, National Cancer Centre Singapore, Singapore. Hauptthemen laut MEDLINE: Neoplasms, Radiology. Die Arbeit wurde bisher 47-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: A*STAR, Singapore Health Services under the Singhealth Duke-NUS Oncology ACP Programme.
Abstract (Originalsprache)
Objective
To assess large language models on their ability to accurately infer cancer disease response from free-text radiology reports. Materials and Methods We assembled 10 602 computed tomography reports from cancer patients seen at a single institution. All reports were classified into: no evidence of disease, partial response, stable disease, or progressive disease. We applied transformer models, a bidirectional long short-term memory model, a convolutional neural network model, and conventional machine learning methods to this task. Data augmentation using sentence permutation with consistency loss as well as prompt-based fine-tuning were used on the best-performing models. Models were validated on a hold-out test set and an external validation set based on Response Evaluation Criteria in Solid Tumors (RECIST) classifications.
Results
The best-performing model was the GatorTron transformer which achieved an accuracy of 0.8916 on the test set and 0.8919 on the RECIST validation set. Data augmentation further improved the accuracy to 0.8976. Prompt-based fine-tuning did not further improve accuracy but was able to reduce the number of training reports to 500 while still achieving good performance.
Discussion
These models could be used by researchers to derive progression-free survival in large datasets. It may also serve as a decision support tool by providing clinicians an automated second opinion of disease response.
Conclusions
Large clinical language models demonstrate potential to infer cancer disease response from radiology reports at scale. Data augmentation techniques are useful to further improve performance. Prompt-based fine-tuning can significantly reduce the size of the training dataset.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Division of Medical Oncology, National Cancer Centre Singapore, Singapore.
- Duke-NUS Medical School, Singapore.
- Department of Computer Science, National University of Singapore, Singapore.
- Institute of Systems Science, National University of Singapore, Singapore.
- Division of Radiation Oncology, National Cancer Centre Singapore, Singapore.
- Department of Diagnostic Radiology, Singapore General Hospital, Singapore.
ORCID-Profile: Ryan Shea Ying Cong Tan, Matthew Chin Heng Chua
Förderung
- A*STAR
- Singapore Health Services under the Singhealth Duke-NUS Oncology ACP Programme
Zitieren
APA
Tan, R. S. Y. C., Lin, Q., Low, G. H., Lin, R., Goh, T. C., Chang, C. C. E., Lee, F. F., Chan, W. Y., Tan, W. C., Tey, H. J., Leong, F. L., Tan, H. Q., Nei, W. L., Chay, W. Y., Tai, D. W. M., Lai, G. G. Y., Cheng, L. T., Wong, F. Y., Chua, M. C. H., …, & Ng, H. T. (2023). Inferring cancer disease response from radiology reports using large language models with data augmentation and prompting. Journal of the American Medical Informatics Association, 30(10), 1657-1664. https://doi.org/10.1093/jamia/ocad133
Vancouver
Tan RSYC, Lin Q, Low GH, Lin R, Goh TC, Chang CCE, et al. Inferring cancer disease response from radiology reports using large language models with data augmentation and prompting. Journal of the American Medical Informatics Association. 2023;30(10):1657-1664. doi:10.1093/jamia/ocad133
BibTeX
@article{Tan2023,
author = {Tan, Ryan Shea Ying Cong and Lin, Qian and Low, Guat Hwa and Lin, Ruixi and Goh, Tzer Chew and Chang, Christopher Chu En and Lee, Fung Fung and Chan, Wei Yin and Tan, Wei Chong and Tey, Han Jieh and Leong, Fun Loon and Tan, Hong Qi and Nei, Wen Long and Chay, Wen Yee and Tai, David Wai Meng and Lai, Gillianne Geet Yi and Cheng, Lionel Tim-Ee and Wong, Fuh Yong and Chua, Matthew Chin Heng and Chua, Melvin Lee Kiang and Tan, Daniel Shao Weng and Thng, Choon Hua and Tan, Iain Bee Huat and Ng, Hwee Tou},
title = {Inferring cancer disease response from radiology reports using large language models with data augmentation and prompting},
journal = {Journal of the American Medical Informatics Association},
year = {2023},
volume = {30},
number = {10},
pages = {1657--1664},
doi = {10.1093/jamia/ocad133}
}
EUVIMED