EUVIMEDEUVIMED entdecken
Open Access

Harnessing Moderate-Sized Language Models for Reliable Patient Data Deidentification in Emergency Department Records: Algorithm Development, Validation, and Implementation Study

Océane Dorémus, Dylan Russon, Benjamin Contrand, Ariel Guerra-Adames, Marta Avalos-Fernandez, Cédric Gil-Jardiné, Emmanuel Lagarde

JMIR AI · 01.04.2025 · 4 · e57828-e57828

DOI: 10.2196/57828 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR AI, online erschienen am 01.04.2025. Verfasst von Océane Dorémus und 6 weiteren Autor:innen. Beteiligte Einrichtung: AHeaD Team, University of Bordeaux, INSERM, BPH, U1219, 146 Rue Léo Saignat, Bordeaux, F-33000, France, 33 5 57 57 15 04. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 46 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The digitization of health care, facilitated by the adoption of electronic health records systems, has revolutionized data-driven medical research and patient care. While this digital transformation offers substantial benefits in health care efficiency and accessibility, it concurrently raises significant concerns over privacy and data security. Initially, the journey toward protecting patient data deidentification saw the transition from rule-based systems to more mixed approaches including machine learning for deidentifying patient data. Subsequently, the emergence of large language models has represented a further opportunity in this domain, offering unparalleled potential for enhancing the accuracy of context-sensitive deidentification. However, despite large language models offering significant potential, the deployment of the most advanced models in hospital environments is frequently hindered by data security issues and the extensive hardware resources required.

Objective

The objective of our study is to design, implement, and evaluate deidentification algorithms using fine-tuned moderate-sized open-source language models, ensuring their suitability for production inference tasks on personal computers.

Methods

We aimed to replace personal identifying information (PII) with generic placeholders or labeling non-PII texts as “ANONYMOUS,” ensuring privacy while preserving textual integrity. Our dataset, derived from over 425,000 clinical notes from the adult emergency department of the Bordeaux University Hospital in France, underwent independent double annotation by 2 experts to create a reference for model validation with 3000 clinical notes randomly selected. Three open-source language models of manageable size were selected for their feasibility in hospital settings: Llama 2 (Meta) 7B, Mistral 7B, and Mixtral 8×7B (Mistral AI). Fine-tuning used the quantized low-rank adaptation technique. Evaluation focused on PII-level (recall, precision, and F 1 -score) and clinical note-level metrics (recall and BLEU [bilingual evaluation understudy] metric), assessing deidentification effectiveness and content preservation.

Results

The generative model Mistral 7B performed the highest with an overall F 1 -score of 0.9673 (vs 0.8750 for Llama 2 and 0.8686 for Mixtral 8×7B). At the clinical notes level, the model’s overall recall was 0.9326 (vs 0.6888 for Llama 2 and 0.6417 for Mixtral 8×7B). This rate increased to 0.9915 when Mistral 7B only deleted names. Four notes of 3000 failed to be fully pseudonymized for names: in 1 case, the nondeleted name belonged to a patient, while in the others, it belonged to medical staff. Beyond the fifth epoch, the BLEU score consistently exceeded 0.9864, indicating no significant text alteration.

Conclusions

Our research underscores the significant capabilities of generative natural language processing models, with Mistral 7B standing out for its superior ability to deidentify clinical texts efficiently. Achieving notable performance metrics, Mistral 7B operates effectively without requiring high-end computational resources. These methods pave the way for a broader availability of pseudonymized clinical texts, enabling their use for research purposes and the optimization of the health care system.

Schlagwörter

Stichwörter der Autor:innen

clinical notesde-identificationelectronic health recordsgeneral data protection regulationlarge language modelmachine learningnatural language processingtransformers

Autor:innen und Institutionen

  • AHeaD Team, University of Bordeaux, INSERM, BPH, U1219, 146 Rue Léo Saignat, Bordeaux, F-33000, France, 33 5 57 57 15 04.
  • SISTM Team, University of Bordeaux, INSERM, INRIA, BPH, U1219, Bordeaux, F-33000, France.
  • Department of Emergency Medicine, Bordeaux University Hospital, Bordeaux, F-33000, France.

Zitieren

APA

Dorémus, O., Russon, D., Contrand, B., Guerra-Adames, A., Avalos-Fernandez, M., Gil-Jardiné, C., & Lagarde, E. (2025). Harnessing Moderate-Sized Language Models for Reliable Patient Data Deidentification in Emergency Department Records: Algorithm Development, Validation, and Implementation Study. JMIR AI, 4, e57828-e57828. https://doi.org/10.2196/57828

Vancouver

Dorémus O, Russon D, Contrand B, Guerra-Adames A, Avalos-Fernandez M, Gil-Jardiné C, et al. Harnessing Moderate-Sized Language Models for Reliable Patient Data Deidentification in Emergency Department Records: Algorithm Development, Validation, and Implementation Study. JMIR AI. 2025;4:e57828-e57828. doi:10.2196/57828

BibTeX
@article{Dormus2025,
  author = {Dorémus, Océane and Russon, Dylan and Contrand, Benjamin and Guerra-Adames, Ariel and Avalos-Fernandez, Marta and Gil-Jardiné, Cédric and Lagarde, Emmanuel},
  title = {Harnessing Moderate-Sized Language Models for Reliable Patient Data Deidentification in Emergency Department Records: Algorithm Development, Validation, and Implementation Study},
  journal = {JMIR AI},
  year = {2025},
  volume = {4},
  pages = {e57828--e57828},
  doi = {10.2196/57828}
}
Kostenlos starten →