Metastasis Extraction From Non–Small Cell Lung Cancer Clinical Notes Using Large Language Model–Based Classification: Retrospective Comparative Study
JMIR Cancer · 16.10.2026 · 12 · e99573
DOI: 10.2196/99573 ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift JMIR Cancer, online erschienen am 16.10.2026. Verfasst von Sweta Balaji und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, United States. Hauptthemen laut MEDLINE: Carcinoma, Non-Small-Cell Lung, Lung Neoplasms. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NLM NIH HHS.
Abstract (Originalsprache)
Background
Identification of metastasis status in non–small cell lung cancer (NSCLC) is a critical part of understanding disease prognosis, treatment courses, trial eligibility, and population-level cancer surveillance. However, metastasis status is inconsistently recorded in structured cancer registry fields, since manual abstraction of clinical notes is often an error-prone process. This challenge highlights an opportunity for leveraging large language models (LLMs) to conduct high-scale metastasis extraction from real-world clinical documentation.
Objective
We conducted a retrospective, multicohort comparative evaluation of 3 distinct LLMs for 2 independent classification tasks: overall metastasis presence at any site and brain/central nervous system (CNS) metastasis presence. We evaluated model performance on 2 independent NSCLC cohorts: one with registry labels and another with manual annotations. We further explored whether our methods could analyze clinical documentation and recover missing information in structured registry fields.
Methods
Patient cohorts were derived from the Winship Cancer Institute. Cohort 1 (n=579 patients; 24,887 notes across 69 note types; 2023-2025) used registry-linked metastasis fields as the reference standard. Cohort 2 (n=22 patients; 644 radiology notes; 2010-2021) was drawn from 2 completed randomized trials and used dual-annotated labels (Cohen κ: 0.93 overall metastasis; 0.88 CNS metastasis) as the reference standard. We fine-tuned the GatorTron-base encoder model for each independent binary classification task. We evaluated MedGemma-27B-text and Llama 3.1-70B using zero-shot prompting. A separate cohort of 675 patients with missing registry labels was used for an exploratory missingness-recovery analysis, validated against annotations of a random subsample. Our best-performing model was used to conduct further site-specific metastasis classification, based on available registry data.
Results
More than half (675/1260, 54%) of initially identified cohort 1 patients had missing or unknown registry metastasis labels. MedGemma demonstrated the best performance in overall metastasis classification (cohort 1: F1-score=0.80; cohort 2 patient-level: F1-score=1.0; cohort 2 note-level: F1-score=0.93). For brain/CNS metastasis, Llama3 performed best in both cohorts (cohort 1: F1-score=0.79; cohort 2 patient-level: F1-score=0.93; cohort 2 note-level: F1-score=0.86). The fine-tuned GatorTron model showed strong performance for classification of overall metastasis in cohort 1 (F1-score=0.72). Error analysis indicated that several model errors reflected incomplete registry labels, ambiguous clinical language, or missing documentation. In the exploratory recovery analysis, model predictions agreed with manual annotations at an accuracy of 0.90 and F1-score of 0.89. Additional site-specific metastasis classification by MedGemma demonstrated high performance for liver metastasis (F1-score=0.82) and bone metastasis (F1-score=0.79).
Conclusions
All models demonstrated relatively high performance. The zero-shot generative models were more robust to nuanced documentation and context-dependent brain/CNS metastasis extraction. The fine-tuned encoder model demonstrated strong classification performance but may have been limited by potential inaccuracies in the registry reference standards during model training. This study further demonstrated the potential of LLMs in recovering plausible structured labels from narrative text, complementing cancer registries for metastasis ascertainment.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, United States.
- Department of Epidemiology, Rollins School of Public Health, Emory University, Atlanta, GA, United States.
- Department of IT and Research Informatics, Winship Cancer Institute, Atlanta, GA, United States.
- Department of Hematology and Medical Oncology, School of Medicine, Emory University, Atlanta, GA, United States.
ORCID-Profile: Sweta Balaji, Kiersten Campbell, Rou-Zhen Chen, Daniel Smith, Matthew A Reyna, Abeed Sarker, Joshua D Wallach, Ravi B Parikh, Selen Bozkurt
Förderung
- NLM NIH HHS
Zitieren
APA
Balaji, S., Campbell, K., Chen, R. Z., Smith, D., Reyna, M. A., Sarker, A., Wallach, J. D., Parikh, R. B., & Bozkurt, S. (2026). Metastasis Extraction From Non–Small Cell Lung Cancer Clinical Notes Using Large Language Model–Based Classification: Retrospective Comparative Study. JMIR Cancer, 12, e99573. https://doi.org/10.2196/99573
Vancouver
Balaji S, Campbell K, Chen RZ, Smith D, Reyna MA, Sarker A, et al. Metastasis Extraction From Non–Small Cell Lung Cancer Clinical Notes Using Large Language Model–Based Classification: Retrospective Comparative Study. JMIR Cancer. 2026;12:e99573. doi:10.2196/99573
BibTeX
@article{Balaji2026,
author = {Balaji, Sweta and Campbell, Kiersten and Chen, Rou-Zhen and Smith, Daniel and Reyna, Matthew A and Sarker, Abeed and Wallach, Joshua D and Parikh, Ravi B and Bozkurt, Selen},
title = {Metastasis Extraction From Non–Small Cell Lung Cancer Clinical Notes Using Large Language Model–Based Classification: Retrospective Comparative Study},
journal = {JMIR Cancer},
year = {2026},
volume = {12},
pages = {e99573},
doi = {10.2196/99573}
}
EUVIMED