Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study
JMIR Cancer · 02.10.2025 · 11 · e72005-e72005
DOI: 10.2196/72005 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Cancer, online erschienen am 02.10.2025. Verfasst von Soheil Hashtarkhani und 8 weiteren Autor:innen. Beteiligte Einrichtung: Center for Biomedical Informatics, Department of Pediatrics, College of Medicine, University of Tennessee Health Science Center, 50 N Dunlap Street, …. Hauptthemen laut MEDLINE: Electronic Health Records, Neoplasms, Natural Language Processing. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 14 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Electronic health records contain inconsistently structured or free-text data, requiring efficient preprocessing to enable predictive health care models. Although artificial intelligence–driven natural language processing tools show promise for automating diagnosis classification, their comparative performance and clinical reliability require systematic evaluation.
Objective
The aim of this study is to evaluate the performance of 4 large language models (GPT-3.5, GPT-4o, Llama 3.2, and Gemini 1.5) and BioBERT in classifying cancer diagnoses from structured and unstructured electronic health records data.
Methods
We analyzed 762 unique diagnoses (326 International Classification of Diseases [ICD] code descriptions, 436 free-text entries) from 3456 records of patients with cancer. Models were tested on their ability to categorize diagnoses into 14 predefined categories. Two oncology experts validated classifications.
Results
BioBERT achieved the highest weighted macro F 1 -score for ICD codes (84.2) and matched GPT-4o in ICD code accuracy (90.8). For free-text diagnoses, GPT-4o outperformed BioBERT in weighted macro F 1 -score (71.8 vs 61.5) and achieved slightly higher accuracy (81.9 vs 81.6). GPT-3.5, Gemini, and Llama showed lower overall performance on both formats. Common misclassification patterns included confusion between metastasis and central nervous system tumors, as well as errors involving ambiguous or overlapping clinical terminology.
Conclusions
Although current performance levels appear sufficient for administrative and research use, reliable clinical applications will require standardized documentation practices alongside robust human oversight for high-stakes decision-making.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Center for Biomedical Informatics, Department of Pediatrics, College of Medicine, University of Tennessee Health Science Center, 50 N Dunlap Street, Memphis, TN, 38103, United States, 1 9012875836.
- University of Tennessee Graduate School of Medicine, Knoxville, TN, United States.
- Departments of Radiation Oncology and Preventive Medicine, College of Medicine, University of Tennessee Health Science Center, Memphis, TN, United States.
ORCID-Profile: Soheil Hashtarkhani, Rezaur Rashid, Christopher L Brett, Lokesh Chinthala, Fekede Asefa Kumsa, Janet A Zink, Robert L Davis, David L Schwartz, Arash Shaban-Nejad
Zitieren
APA
Hashtarkhani, S., Rashid, R., Brett, C. L., Chinthala, L., Kumsa, F. A., Zink, J. A., Davis, R. L., Schwartz, D. L., & Shaban-Nejad, A. (2025). Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study. JMIR Cancer, 11, e72005-e72005. https://doi.org/10.2196/72005
Vancouver
Hashtarkhani S, Rashid R, Brett CL, Chinthala L, Kumsa FA, Zink JA, et al. Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study. JMIR Cancer. 2025;11:e72005-e72005. doi:10.2196/72005
BibTeX
@article{Hashtarkhani2025,
author = {Hashtarkhani, Soheil and Rashid, Rezaur and Brett, Christopher L and Chinthala, Lokesh and Kumsa, Fekede Asefa and Zink, Janet A and Davis, Robert L and Schwartz, David L and Shaban-Nejad, Arash},
title = {Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study},
journal = {JMIR Cancer},
year = {2025},
volume = {11},
pages = {e72005--e72005},
doi = {10.2196/72005}
}
EUVIMED