EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

Using Synthetic Health Care Data to Leverage Large Language Models for Named Entity Recognition: Development and Validation Study

Hendrik Šuvalov iD, Mihkel Lepson iD, Veronika Kukk iD, Maria Malk iD, Neeme Ilves iD, Hele-Andra Kuulmets iD, Raivo Kolde iD

Journal of Medical Internet Research · 18.03.2025 · 27 · e66279

DOI: 10.2196/66279 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 18.03.2025. Verfasst von Hendrik Šuvalov und 6 weiteren Autor:innen. Beteiligte Einrichtung: Institute of Computer Science, University of Tartu, Tartu, Estonia. Hauptthemen laut MEDLINE: Natural Language Processing, Data Mining, Electronic Health Records. Die Arbeit wurde bisher 18-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Named entity recognition (NER) plays a vital role in extracting critical medical entities from health care records, facilitating applications such as clinical decision support and data mining. Developing robust NER models for low-resource languages, such as Estonian, remains a challenge due to the scarcity of annotated data and domain-specific pretrained models. Large language models (LLMs) have proven to be promising in understanding text from any language or domain.

Objective

This study addresses the development of medical NER models for low-resource languages, specifically Estonian. We propose a novel approach by generating synthetic health care data and using LLMs to annotate them. These synthetic data are then used to train a high-performing NER model, which is applied to real-world medical texts, preserving patient data privacy.

Methods

Our approach to overcoming the shortage of annotated Estonian health care texts involves a three-step pipeline: (1) synthetic health care data are generated using a locally trained GPT-2 model on Estonian medical records, (2) the synthetic data are annotated with LLMs, specifically GPT-3.5-Turbo and GPT-4, and (3) the annotated synthetic data are then used to fine-tune an NER model, which is later tested on real-world medical data. This paper compares the performance of different prompts; assesses the impact of GPT-3.5-Turbo, GPT-4, and a local LLM; and explores the relationship between the amount of annotated synthetic data and model performance.

Results

The proposed methodology demonstrates significant potential in extracting named entities from real-world medical texts. Our top-performing setup achieved an F1-score of 0.69 for drug extraction and 0.38 for procedure extraction. These results indicate a strong performance in recognizing certain entity types while highlighting the complexity of extracting procedures.

Conclusions

This paper demonstrates a successful approach to leveraging LLMs for training NER models using synthetic data, effectively preserving patient privacy. By avoiding reliance on human-annotated data, our method shows promise in developing models for low-resource languages, such as Estonian. Future work will focus on refining the synthetic data generation and expanding the method’s applicability to other domains and languages.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingData MiningElectronic Health RecordsHumansEstoniaLarge Language Models

Stichwörter der Autor:innen

EstonianLLMNERNLPannotated dataartificial intelligenceclinical decision supportdata annotationdata mininghealth care datalanguage modellarge language modelmachine learningmedical entitynamed entity recognition

Autor:innen und Institutionen

  • Institute of Computer Science, University of Tartu, Tartu, Estonia.

ORCID-Profile: Hendrik Šuvalov, Mihkel Lepson, Veronika Kukk, Maria Malk, Neeme Ilves, Hele-Andra Kuulmets, Raivo Kolde

Zitieren

APA

Šuvalov, H., Lepson, M., Kukk, V., Malk, M., Ilves, N., Kuulmets, H. A., & Kolde, R. (2025). Using Synthetic Health Care Data to Leverage Large Language Models for Named Entity Recognition: Development and Validation Study. Journal of Medical Internet Research, 27, e66279. https://doi.org/10.2196/66279

Vancouver

Šuvalov H, Lepson M, Kukk V, Malk M, Ilves N, Kuulmets HA, et al. Using Synthetic Health Care Data to Leverage Large Language Models for Named Entity Recognition: Development and Validation Study. Journal of Medical Internet Research. 2025;27:e66279. doi:10.2196/66279

BibTeX
@article{uvalov2025,
  author = {Šuvalov, Hendrik and Lepson, Mihkel and Kukk, Veronika and Malk, Maria and Ilves, Neeme and Kuulmets, Hele-Andra and Kolde, Raivo},
  title = {Using Synthetic Health Care Data to Leverage Large Language Models for Named Entity Recognition: Development and Validation Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e66279},
  doi = {10.2196/66279}
}
Kostenlos starten →