EUVIMEDObjevte EUVIMED
Open access

Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study

Soheil Hashtarkhani iD, Rezaur Rashid iD, Christopher L Brett iD, Lokesh Chinthala iD, Fekede Asefa Kumsa iD, Janet A Zink iD, Robert L Davis iD, David L Schwartz iD, Arash Shaban-Nejad iD

JMIR Cancer · 2025-10-02 · 11 · e72005-e72005

DOI: 10.2196/72005 ↗

At a glance

Journal article in JMIR Cancer, published online on 2025-10-02. Written by Soheil Hashtarkhani and 8 co-authors. Institution: Center for Biomedical Informatics, Department of Pediatrics, College of Medicine, University of Tennessee Health Science Center, 50 N Dunlap Street, …. Major MEDLINE topics: Electronic Health Records, Neoplasms, Natural Language Processing. Cited 7 times so far (Crossref); 14 references. The full text is freely available in PubMed Central.

AbstraktSubjectsBibliografické údajeCiteRelated studies

Abstract (original language)

Background

Electronic health records contain inconsistently structured or free-text data, requiring efficient preprocessing to enable predictive health care models. Although artificial intelligence–driven natural language processing tools show promise for automating diagnosis classification, their comparative performance and clinical reliability require systematic evaluation.

Objective

The aim of this study is to evaluate the performance of 4 large language models (GPT-3.5, GPT-4o, Llama 3.2, and Gemini 1.5) and BioBERT in classifying cancer diagnoses from structured and unstructured electronic health records data.

Methods

We analyzed 762 unique diagnoses (326 International Classification of Diseases [ICD] code descriptions, 436 free-text entries) from 3456 records of patients with cancer. Models were tested on their ability to categorize diagnoses into 14 predefined categories. Two oncology experts validated classifications.

Results

BioBERT achieved the highest weighted macro F 1 -score for ICD codes (84.2) and matched GPT-4o in ICD code accuracy (90.8). For free-text diagnoses, GPT-4o outperformed BioBERT in weighted macro F 1 -score (71.8 vs 61.5) and achieved slightly higher accuracy (81.9 vs 81.6). GPT-3.5, Gemini, and Llama showed lower overall performance on both formats. Common misclassification patterns included confusion between metastasis and central nervous system tumors, as well as errors involving ambiguous or overlapping clinical terminology.

Conclusions

Although current performance levels appear sufficient for administrative and research use, reliable clinical applications will require standardized documentation practices alongside robust human oversight for high-stakes decision-making.

Subjects

MeSH terms (MEDLINE)

Electronic Health RecordsNeoplasmsNatural Language ProcessingHumansInternational Classification of DiseasesReproducibility of ResultsArtificial IntelligenceLarge Language Models

Author keywords

BioBERTbidirectional encoder representations from transformerscancer diagnosiselectronic health recordlarge language modelnatural language processing

Authors and institutions

  • Center for Biomedical Informatics, Department of Pediatrics, College of Medicine, University of Tennessee Health Science Center, 50 N Dunlap Street, Memphis, TN, 38103, United States, 1 9012875836.
  • University of Tennessee Graduate School of Medicine, Knoxville, TN, United States.
  • Departments of Radiation Oncology and Preventive Medicine, College of Medicine, University of Tennessee Health Science Center, Memphis, TN, United States.

ORCID profiles: Soheil Hashtarkhani, Rezaur Rashid, Christopher L Brett, Lokesh Chinthala, Fekede Asefa Kumsa, Janet A Zink, Robert L Davis, David L Schwartz, Arash Shaban-Nejad

Cite

APA

Hashtarkhani, S., Rashid, R., Brett, C. L., Chinthala, L., Kumsa, F. A., Zink, J. A., Davis, R. L., Schwartz, D. L., & Shaban-Nejad, A. (2025). Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study. JMIR Cancer, 11, e72005-e72005. https://doi.org/10.2196/72005

Vancouver

Hashtarkhani S, Rashid R, Brett CL, Chinthala L, Kumsa FA, Zink JA, et al. Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study. JMIR Cancer. 2025;11:e72005-e72005. doi:10.2196/72005

BibTeX
@article{Hashtarkhani2025,
  author = {Hashtarkhani, Soheil and Rashid, Rezaur and Brett, Christopher L and Chinthala, Lokesh and Kumsa, Fekede Asefa and Zink, Janet A and Davis, Robert L and Schwartz, David L and Shaban-Nejad, Arash},
  title = {Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study},
  journal = {JMIR Cancer},
  year = {2025},
  volume = {11},
  pages = {e72005--e72005},
  doi = {10.2196/72005}
}
Start for free →