EUVIMEDEUVIMED entdecken
Open Access

Unlocking the Secrets Behind Advanced Artificial Intelligence Language Models in Deidentifying Chinese-English Mixed Clinical Text: Development and Validation Study

You-Qian Lee iD, Ching-Tai Chen iD, Chien-Chang Chen iD, Chung-Hong Lee iD, Peitsz Chen iD, Chi-Shin Wu iD, Hong-Jie Dai iD

Journal of Medical Internet Research · 2024

DOI: 10.2196/48443 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, 2024. Verfasst von You-Qian Lee und 6 weiteren Autor:innen. Beteiligte Einrichtung: Dialogue System Technical Department, Intelligent Robot, Asustek Computer Inc, Taipei, Taiwan. Hauptthemen laut MEDLINE: Artificial Intelligence, Electronic Health Records. Die Arbeit wurde bisher 20-mal zitiert (Crossref) und nennt 60 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The widespread use of electronic health records in the clinical and biomedical fields makes the removal of protected health information (PHI) essential to maintain privacy. However, a significant portion of information is recorded in unstructured textual forms, posing a challenge for deidentification. In multilingual countries, medical records could be written in a mixture of more than one language, referred to as code mixing. Most current clinical natural language processing techniques are designed for monolingual text, and there is a need to address the deidentification of code-mixed text.

Objective

The aim of this study was to investigate the effectiveness and underlying mechanism of fine-tuned pretrained language models (PLMs) in identifying PHI in the code-mixed context. Additionally, we aimed to evaluate the potential of prompting large language models (LLMs) for recognizing PHI in a zero-shot manner.

Methods

We compiled the first clinical code-mixed deidentification data set consisting of text written in Chinese and English. We explored the effectiveness of fine-tuned PLMs for recognizing PHI in code-mixed content, with a focus on whether PLMs exploit naming regularity and mention coverage to achieve superior performance, by probing the developed models’ outputs to examine their decision-making process. Furthermore, we investigated the potential of prompt-based in-context learning of LLMs for recognizing PHI in code-mixed text.

Results

The developed methods were evaluated on a code-mixed deidentification corpus of 1700 discharge summaries. We observed that different PHI types had preferences in their occurrences within the different types of language-mixed sentences, and PLMs could effectively recognize PHI by exploiting the learned name regularity. However, the models may exhibit suboptimal results when regularity is weak or mentions contain unknown words that the representations cannot generate well. We also found that the availability of code-mixed training instances is essential for the model’s performance. Furthermore, the LLM-based deidentification method was a feasible and appealing approach that can be controlled and enhanced through natural language prompts.

Conclusions

The study contributes to understanding the underlying mechanism of PLMs in addressing the deidentification process in the code-mixed context and highlights the significance of incorporating code-mixed training instances into the model training phase. To support the advancement of research, we created a manipulated subset of the resynthesized data set available for research purposes. Based on the compiled data set, we found that the LLM-based deidentification method is a feasible approach, but carefully crafted prompts are essential to avoid unwanted output. However, the use of such methods in the hospital setting requires careful consideration of data security and privacy concerns. Further research could explore the augmentation of PLMs and LLMs with external knowledge to improve their strength in recognizing rare PHI.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceElectronic Health RecordsHumansNatural Language ProcessingPrivacyChina

Stichwörter der Autor:innen

ChatGPTcode mixingdeidentificationelectronic health recordlarge language modelpretrained language model

Autor:innen und Institutionen

  • Dialogue System Technical Department, Intelligent Robot, Asustek Computer Inc, Taipei, Taiwan.
  • Intelligent System Laboratory, Department of Electrical Engineering, College of Electrical Engineering and Computer Science, National Kaohsiung University of Science and Technology, Kaohsiung, Taiwan.
  • Department of Bioinformatics and Medical Engineering, Asia University, Taichung, Taiwan.
  • Center for Precision Health Research, Asia University, Taichung, Taiwan.
  • Electromagnetic Sensing Control and AI Computing System Laboratory, Department of Electrical Engineering, College of Electrical Engineering and Computer Science, National Kaohsiung University of Science and Technology,…
  • Knowledge Discovery and Data Mining Lab, Department of Electrical Engineering, College of Electrical Engineering and Computer Science, National Kaohsiung University of Science and Technology, Kaohsiung, Taiwan.

ORCID-Profile: You-Qian Lee, Ching-Tai Chen, Chien-Chang Chen, Chung-Hong Lee, Peitsz Chen, Chi-Shin Wu, Hong-Jie Dai

Zitieren

APA

Lee, Y. Q., Chen, C. T., Chen, C. C., Lee, C. H., Chen, P., Wu, C. S., & Dai, H. J. (2024). Unlocking the Secrets Behind Advanced Artificial Intelligence Language Models in Deidentifying Chinese-English Mixed Clinical Text: Development and Validation Study. Journal of Medical Internet Research. https://doi.org/10.2196/48443

Vancouver

Lee YQ, Chen CT, Chen CC, Lee CH, Chen P, Wu CS, et al. Unlocking the Secrets Behind Advanced Artificial Intelligence Language Models in Deidentifying Chinese-English Mixed Clinical Text: Development and Validation Study. Journal of Medical Internet Research. 2024. doi:10.2196/48443

BibTeX
@article{Lee2024,
  author = {Lee, You-Qian and Chen, Ching-Tai and Chen, Chien-Chang and Lee, Chung-Hong and Chen, Peitsz and Wu, Chi-Shin and Dai, Hong-Jie},
  title = {Unlocking the Secrets Behind Advanced Artificial Intelligence Language Models in Deidentifying Chinese-English Mixed Clinical Text: Development and Validation Study},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  doi = {10.2196/48443}
}
Kostenlos starten →