EUVIMEDOpdag EUVIMED
ValidierungsstudieOpen access

Using Synthetic Health Care Data to Leverage Large Language Models for Named Entity Recognition: Development and Validation Study

Hendrik Šuvalov iD, Mihkel Lepson iD, Veronika Kukk iD, Maria Malk iD, Neeme Ilves iD, Hele-Andra Kuulmets iD, Raivo Kolde iD

Journal of Medical Internet Research · 2025-03-18 · 27 · e66279

DOI: 10.2196/66279 ↗

At a glance

Validierungsstudie in Journal of Medical Internet Research, published online on 2025-03-18. Written by Hendrik Šuvalov and 6 co-authors. Institution: Institute of Computer Science, University of Tartu, Tartu, Estonia. Major MEDLINE topics: Natural Language Processing, Data Mining, Electronic Health Records. Cited 18 times so far (Crossref); 35 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Named entity recognition (NER) plays a vital role in extracting critical medical entities from health care records, facilitating applications such as clinical decision support and data mining. Developing robust NER models for low-resource languages, such as Estonian, remains a challenge due to the scarcity of annotated data and domain-specific pretrained models. Large language models (LLMs) have proven to be promising in understanding text from any language or domain.

Objective

This study addresses the development of medical NER models for low-resource languages, specifically Estonian. We propose a novel approach by generating synthetic health care data and using LLMs to annotate them. These synthetic data are then used to train a high-performing NER model, which is applied to real-world medical texts, preserving patient data privacy.

Methods

Our approach to overcoming the shortage of annotated Estonian health care texts involves a three-step pipeline: (1) synthetic health care data are generated using a locally trained GPT-2 model on Estonian medical records, (2) the synthetic data are annotated with LLMs, specifically GPT-3.5-Turbo and GPT-4, and (3) the annotated synthetic data are then used to fine-tune an NER model, which is later tested on real-world medical data. This paper compares the performance of different prompts; assesses the impact of GPT-3.5-Turbo, GPT-4, and a local LLM; and explores the relationship between the amount of annotated synthetic data and model performance.

Results

The proposed methodology demonstrates significant potential in extracting named entities from real-world medical texts. Our top-performing setup achieved an F1-score of 0.69 for drug extraction and 0.38 for procedure extraction. These results indicate a strong performance in recognizing certain entity types while highlighting the complexity of extracting procedures.

Conclusions

This paper demonstrates a successful approach to leveraging LLMs for training NER models using synthetic data, effectively preserving patient privacy. By avoiding reliance on human-annotated data, our method shows promise in developing models for low-resource languages, such as Estonian. Future work will focus on refining the synthetic data generation and expanding the method’s applicability to other domains and languages.

Subjects

MeSH terms (MEDLINE)

Natural Language ProcessingData MiningElectronic Health RecordsHumansEstoniaLarge Language Models

Author keywords

EstonianLLMNERNLPannotated dataartificial intelligenceclinical decision supportdata annotationdata mininghealth care datalanguage modellarge language modelmachine learningmedical entitynamed entity recognition

Authors and institutions

  • Institute of Computer Science, University of Tartu, Tartu, Estonia.

ORCID profiles: Hendrik Šuvalov, Mihkel Lepson, Veronika Kukk, Maria Malk, Neeme Ilves, Hele-Andra Kuulmets, Raivo Kolde

Cite

APA

Šuvalov, H., Lepson, M., Kukk, V., Malk, M., Ilves, N., Kuulmets, H. A., & Kolde, R. (2025). Using Synthetic Health Care Data to Leverage Large Language Models for Named Entity Recognition: Development and Validation Study. Journal of Medical Internet Research, 27, e66279. https://doi.org/10.2196/66279

Vancouver

Šuvalov H, Lepson M, Kukk V, Malk M, Ilves N, Kuulmets HA, et al. Using Synthetic Health Care Data to Leverage Large Language Models for Named Entity Recognition: Development and Validation Study. Journal of Medical Internet Research. 2025;27:e66279. doi:10.2196/66279

BibTeX
@article{uvalov2025,
  author = {Šuvalov, Hendrik and Lepson, Mihkel and Kukk, Veronika and Malk, Maria and Ilves, Neeme and Kuulmets, Hele-Andra and Kolde, Raivo},
  title = {Using Synthetic Health Care Data to Leverage Large Language Models for Named Entity Recognition: Development and Validation Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e66279},
  doi = {10.2196/66279}
}
Start for free →