EUVIMEDOpdag EUVIMED
Open access

The Transformative Potential of Large Language Models in Mining Electronic Health Records Data: Content Analysis

Amadeo Jesus Wals Zurita iD, Hector Miras Del Rio iD, Nerea Ugarte Ruiz de Aguirre iD, Cristina Nebrera Navarro iD, Maria Rubio Jimenez iD, David Muñoz Carmona iD, Carlos Miguez Sanchez iD

JMIR Medical Informatics · 2025-01-02 · 13 · e58457

DOI: 10.2196/58457 ↗

At a glance

Journal article in JMIR Medical Informatics, published online on 2025-01-02. Written by Amadeo Jesus Wals Zurita and 6 co-authors. Institution: Servicio Oncologia Radioterápica, Hospital Universitario Virgen Macarena, Andalusian Health Service, Seville, Spain. Major MEDLINE topics: Data Mining, Electronic Health Records. Cited 21 times so far (Crossref); 24 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

In this study, we evaluate the accuracy, efficiency, and cost-effectiveness of large language models in extracting and structuring information from free-text clinical reports, particularly in identifying and classifying patient comorbidities within oncology electronic health records. We specifically compare the performance of gpt-3.5-turbo-1106 and gpt-4-1106-preview models against that of specialized human evaluators.

Objective

We specifically compare the performance of gpt-3.5-turbo-1106 and gpt-4-1106-preview models against that of specialized human evaluators.

Methods

We implemented a script using the OpenAI application programming interface to extract structured information in JavaScript object notation format from comorbidities reported in 250 personal history reports. These reports were manually reviewed in batches of 50 by 5 specialists in radiation oncology. We compared the results using metrics such as sensitivity, specificity, precision, accuracy, F-value, κ index, and the McNemar test, in addition to examining the common causes of errors in both humans and generative pretrained transformer (GPT) models.

Results

The GPT-3.5 model exhibited slightly lower performance compared to physicians across all metrics, though the differences were not statistically significant (McNemar test, P=.79). GPT-4 demonstrated clear superiority in several key metrics (McNemar test, P

Subjects

MeSH terms (MEDLINE)

Data MiningElectronic Health RecordsHumansNatural Language ProcessingComorbidity

Author keywords

ChatGPTEHRLLMsdata miningelectronic health recordlarge language modelsoncologyradiotherapy

Authors and institutions

  • Servicio Oncologia Radioterápica, Hospital Universitario Virgen Macarena, Andalusian Health Service, Seville, Spain.

ORCID profiles: Amadeo Jesus Wals Zurita, Hector Miras Del Rio, Nerea Ugarte Ruiz de Aguirre, Cristina Nebrera Navarro, Maria Rubio Jimenez, David Muñoz Carmona, Carlos Miguez Sanchez

Cite

APA

Wals Zurita, A. J., Miras Del Rio, H., Ugarte Ruiz de Aguirre, N., Nebrera Navarro, C., Rubio Jimenez, M., Muñoz Carmona, D., & Miguez Sanchez, C. (2025). The Transformative Potential of Large Language Models in Mining Electronic Health Records Data: Content Analysis. JMIR Medical Informatics, 13, e58457. https://doi.org/10.2196/58457

Vancouver

Wals Zurita AJ, Miras Del Rio H, Ugarte Ruiz de Aguirre N, Nebrera Navarro C, Rubio Jimenez M, Muñoz Carmona D, et al. The Transformative Potential of Large Language Models in Mining Electronic Health Records Data: Content Analysis. JMIR Medical Informatics. 2025;13:e58457. doi:10.2196/58457

BibTeX
@article{WalsZurita2025,
  author = {Wals Zurita, Amadeo Jesus and Miras Del Rio, Hector and Ugarte Ruiz de Aguirre, Nerea and Nebrera Navarro, Cristina and Rubio Jimenez, Maria and Muñoz Carmona, David and Miguez Sanchez, Carlos},
  title = {The Transformative Potential of Large Language Models in Mining Electronic Health Records Data: Content Analysis},
  journal = {JMIR Medical Informatics},
  year = {2025},
  volume = {13},
  pages = {e58457},
  doi = {10.2196/58457}
}
Start for free →