Automated anonymization of radiology reports: comparison of publicly available natural language processing and large language models
European Radiology · 2024
DOI: 10.1007/s00330-024-11148-x ↗Kurz gefasst
Vergleichsstudie in der Zeitschrift European Radiology, 2024. Verfasst von Marcel C Langenbach und 7 weiteren Autor:innen. Beteiligte Einrichtung: Cardiovascular Imaging Research Center, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA. mlangenbach@mgh.harvard.edu. Hauptthemen laut MEDLINE: Natural Language Processing, Data Anonymization, Radiology Information Systems. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 20 Literaturangaben. Gefördert von: Deutsche Forschungsgemeinschaft.
Abstract (Originalsprache)
PURPOSE: Medical reports, governed by HIPAA regulations, contain personal health information (PHI), restricting secondary data use. Utilizing natural language processing (NLP) and large language models (LLM), we sought to employ publicly available methods to automatically anonymize PHI in free-text radiology reports. MATERIALS AND METHODS: We compared two publicly available rule-based NLP models (spaCy; NLPac, accuracy-optimized; NLPsp, speed-optimized; iteratively improved on 400 free-text CT-reports (test set)) and one offline LLM approach (LLM-model, LLaMa-2, Meta-AI) for PHI-anonymization. The three models were tested on 100 randomly selected chest CT reports. Two investigators assessed the anonymization of occurring PHI entities and whether clinical information was removed. Subsequently, precision, recall, and F1 scores were calculated. RESULTS: NLPac and NLPsp successfully removed all instances of dates (n = 333), medical record numbers (MRN) (n = 6), and accession numbers (ACC) (n = 92). The LLM model removed all MRNs, 96% of ACCs, and 32% of dates. NLPac was most consistent with a perfect F1-score of 1.00, followed by NLPsp with lower precision (0.86) and F1-score (0.92) for dates. The LLM model had perfect precision for MRNs, ACCs, and dates but the lowest recall for ACC (0.96) and dates (0.52), corresponding F1 scores of 0.98 and 0.68, respectively. Names were removed completely or majorly (i.e., one first or family name non-anonymized) in 100% (NLPac), 72% (NLPsp), and 90% (LLM-model). Importantly, NLPac and NLPsp did not remove medical information, while the LLM model did in 10% (n = 10). CONCLUSION: Pre-trained NLP models can effectively anonymize free-text radiology reports, while anonymization with the LLM model is more prone to deleting medical information. KEY POINTS: Question This study compares NLP and locally hosted LLM techniques to ensure PHI anonymization without losing clinical information. Findings Pre-trained NLP models effectively anonymized radiology reports without removing clinical data, while a locally hosted LLM was less reliable, risking the loss of important information. Clinical relevance Fast, reliable, automated anonymization of PHI from radiology reports enables HIPAA-compliant secondary use, facilitating advanced applications like LLM-driven radiology analysis while ensuring ethical handling of sensitive patient data.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Cardiovascular Imaging Research Center, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA. mlangenbach@mgh.harvard.edu.
- Institute for Diagnostic and Interventional Radiology, University Hospital Cologne, Cologne, Germany. mlangenbach@mgh.harvard.edu.
- Cardiovascular Imaging Research Center, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
- Radiology and Nuclear Medicine, CARIM & GROW, Maastricht University, Maastricht, The Netherlands.
- Institute for Diagnostic and Interventional Radiology, University Hospital Cologne, Cologne, Germany.
- Department of Cardiology, Leiden University Medical Center, Leiden, The Netherlands.
Förderung
- Deutsche Forschungsgemeinschaft
Zitieren
APA
Langenbach, M. C., Foldyna, B., Hadzic, I., Langenbach, I. L., Raghu, V. K., Lu, M. T., Neilan, T. G., & Heemelaar, J. C. (2024). Automated anonymization of radiology reports: comparison of publicly available natural language processing and large language models. European Radiology. https://doi.org/10.1007/s00330-024-11148-x
Vancouver
Langenbach MC, Foldyna B, Hadzic I, Langenbach IL, Raghu VK, Lu MT, et al. Automated anonymization of radiology reports: comparison of publicly available natural language processing and large language models. European Radiology. 2024. doi:10.1007/s00330-024-11148-x
BibTeX
@article{Langenbach2024,
author = {Langenbach, Marcel C and Foldyna, Borek and Hadzic, Ibrahim and Langenbach, Isabel L and Raghu, Vineet K and Lu, Michael T and Neilan, Tomas G and Heemelaar, Julius C},
title = {Automated anonymization of radiology reports: comparison of publicly available natural language processing and large language models},
journal = {European Radiology},
year = {2024},
doi = {10.1007/s00330-024-11148-x}
}
EUVIMED