EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Extracting epilepsy‐related information from unstructured clinic letters using large language models

Shichao Fang iD, Ben Holgate, Anthony Shek, Joel S Winston iD, Matthew McWilliam, Pedro F Viana iD, James T Teo, Mark P Richardson

Epilepsia · 2025

DOI: 10.1111/epi.18475 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Epilepsia, 2025. Verfasst von Shichao Fang und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Basic & Clinical Neuroscience, King's College London, London, UK. Hauptthemen laut MEDLINE: Epilepsy, Electronic Health Records, Natural Language Processing, Data Mining. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 66 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Angelini Pharma, Epilepsy Research Institute UK.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

The emergence of large language models (LLMs) and the increasing prevalence of electronic health records (EHRs) present significant opportunities for advancing health care research and practice. However, research that compares and applies LLMs to extract key epilepsy‐related information from unstructured medical free text is under‐explored. This study fills this gap by comparing and applying different open‐source LLMs and methods to extract epilepsy information from unstructured clinic letters, thereby optimizing EHRs as a resource for the benefit of epilepsy research. We also highlight some limitations of LLMs.

Methods

Employing a dataset of 280 annotated clinic letters from King's College Hospital, we explored the efficacy of open‐source LLMs (Llama and Mistral series) for extracting key epilepsy‐related information, including epilepsy type, seizure type, current anti‐seizure medications (ASMs), and associated symptoms. The study used various extraction methods, including direct extraction, summarized extraction, and contextualized extraction, complemented by role‐prompting and few‐shot prompting techniques. Performance was evaluated against a gold standard dataset, and was also compared to advanced fine‐tuned models and human annotations.

Results

Llama 2 13b (a 13‐billion‐parameter LLM developed by Meta) demonstrated superior extraction capabilities across tasks by consistently outperforming other LLMs (F1 = .80 in epilepsy‐type extraction, F1 = .76 in seizure‐type extraction, and F1 = .90 in current ASMs extraction). Here, F1 score is a balanced metric indicating the model's accuracy in correctly identifying relevant information without excessive false positives. The study highlights the direct extraction showing consistent high performance. Comparative analysis showed that LLMs outperformed current approaches like MedCAT (Medical Concept Annotation Tool) in extracting epilepsy‐related information (.2 higher in F1). Significance The results affirm the potential of LLMs in medical information extraction relating to epilepsy, offering insights into leveraging these models for detailed and accurate data extraction from unstructured texts. The study underscores the importance of method selection in optimizing extraction performance and suggests a promising avenue for enhancing medical research and patient care through advanced natural language processing technologies.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

EpilepsyElectronic Health RecordsNatural Language ProcessingData MiningLanguageInformation Storage and RetrievalHumansLarge Language Models

Stichwörter der Autor:innen

electronic health recordsepilepsylarge language modelsnatural language processing

Autor:innen und Institutionen

  • Department of Basic & Clinical Neuroscience, King's College London, London, UK.
  • King's College Hospital NHS Foundation Trust, London, UK.
  • Guy's and St Thomas' NHS Foundation Trust, London, UK.

ORCID-Profile: Shichao Fang, Joel S Winston, Pedro F Viana

Förderung

  • Angelini Pharma
  • Epilepsy Research Institute UK

Zitieren

APA

Fang, S., Holgate, B., Shek, A., Winston, J. S., McWilliam, M., Viana, P. F., Teo, J. T., & Richardson, M. P. (2025). Extracting epilepsy‐related information from unstructured clinic letters using large language models. Epilepsia. https://doi.org/10.1111/epi.18475

Vancouver

Fang S, Holgate B, Shek A, Winston JS, McWilliam M, Viana PF, et al. Extracting epilepsy‐related information from unstructured clinic letters using large language models. Epilepsia. 2025. doi:10.1111/epi.18475

BibTeX
@article{Fang2025,
  author = {Fang, Shichao and Holgate, Ben and Shek, Anthony and Winston, Joel S and McWilliam, Matthew and Viana, Pedro F and Teo, James T and Richardson, Mark P},
  title = {Extracting epilepsy‐related information from unstructured clinic letters using large language models},
  journal = {Epilepsia},
  year = {2025},
  doi = {10.1111/epi.18475}
}
Kostenlos starten →