EUVIMEDEUVIMED entdecken
Open Access

Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study

Soheil Hashtarkhani iD, Rezaur Rashid iD, Christopher L Brett iD, Lokesh Chinthala iD, Fekede Asefa Kumsa iD, Janet A Zink iD, Robert L Davis iD, David L Schwartz iD, Arash Shaban-Nejad iD

JMIR Cancer · 02.10.2025 · 11 · e72005-e72005

DOI: 10.2196/72005 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Cancer, online erschienen am 02.10.2025. Verfasst von Soheil Hashtarkhani und 8 weiteren Autor:innen. Beteiligte Einrichtung: Center for Biomedical Informatics, Department of Pediatrics, College of Medicine, University of Tennessee Health Science Center, 50 N Dunlap Street, …. Hauptthemen laut MEDLINE: Electronic Health Records, Neoplasms, Natural Language Processing. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 14 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Electronic health records contain inconsistently structured or free-text data, requiring efficient preprocessing to enable predictive health care models. Although artificial intelligence–driven natural language processing tools show promise for automating diagnosis classification, their comparative performance and clinical reliability require systematic evaluation.

Objective

The aim of this study is to evaluate the performance of 4 large language models (GPT-3.5, GPT-4o, Llama 3.2, and Gemini 1.5) and BioBERT in classifying cancer diagnoses from structured and unstructured electronic health records data.

Methods

We analyzed 762 unique diagnoses (326 International Classification of Diseases [ICD] code descriptions, 436 free-text entries) from 3456 records of patients with cancer. Models were tested on their ability to categorize diagnoses into 14 predefined categories. Two oncology experts validated classifications.

Results

BioBERT achieved the highest weighted macro F 1 -score for ICD codes (84.2) and matched GPT-4o in ICD code accuracy (90.8). For free-text diagnoses, GPT-4o outperformed BioBERT in weighted macro F 1 -score (71.8 vs 61.5) and achieved slightly higher accuracy (81.9 vs 81.6). GPT-3.5, Gemini, and Llama showed lower overall performance on both formats. Common misclassification patterns included confusion between metastasis and central nervous system tumors, as well as errors involving ambiguous or overlapping clinical terminology.

Conclusions

Although current performance levels appear sufficient for administrative and research use, reliable clinical applications will require standardized documentation practices alongside robust human oversight for high-stakes decision-making.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Electronic Health RecordsNeoplasmsNatural Language ProcessingHumansInternational Classification of DiseasesReproducibility of ResultsArtificial IntelligenceLarge Language Models

Stichwörter der Autor:innen

BioBERTbidirectional encoder representations from transformerscancer diagnosiselectronic health recordlarge language modelnatural language processing

Autor:innen und Institutionen

  • Center for Biomedical Informatics, Department of Pediatrics, College of Medicine, University of Tennessee Health Science Center, 50 N Dunlap Street, Memphis, TN, 38103, United States, 1 9012875836.
  • University of Tennessee Graduate School of Medicine, Knoxville, TN, United States.
  • Departments of Radiation Oncology and Preventive Medicine, College of Medicine, University of Tennessee Health Science Center, Memphis, TN, United States.

ORCID-Profile: Soheil Hashtarkhani, Rezaur Rashid, Christopher L Brett, Lokesh Chinthala, Fekede Asefa Kumsa, Janet A Zink, Robert L Davis, David L Schwartz, Arash Shaban-Nejad

Zitieren

APA

Hashtarkhani, S., Rashid, R., Brett, C. L., Chinthala, L., Kumsa, F. A., Zink, J. A., Davis, R. L., Schwartz, D. L., & Shaban-Nejad, A. (2025). Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study. JMIR Cancer, 11, e72005-e72005. https://doi.org/10.2196/72005

Vancouver

Hashtarkhani S, Rashid R, Brett CL, Chinthala L, Kumsa FA, Zink JA, et al. Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study. JMIR Cancer. 2025;11:e72005-e72005. doi:10.2196/72005

BibTeX
@article{Hashtarkhani2025,
  author = {Hashtarkhani, Soheil and Rashid, Rezaur and Brett, Christopher L and Chinthala, Lokesh and Kumsa, Fekede Asefa and Zink, Janet A and Davis, Robert L and Schwartz, David L and Shaban-Nejad, Arash},
  title = {Cancer Diagnosis Categorization in Electronic Health Records Using Large Language Models and BioBERT: Model Performance Evaluation Study},
  journal = {JMIR Cancer},
  year = {2025},
  volume = {11},
  pages = {e72005--e72005},
  doi = {10.2196/72005}
}
Kostenlos starten →