EUVIMEDEUVIMED entdecken
Open Access

Multifaceted Natural Language Processing Task–Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation

Kyungmo Kim iD, Seongkeun Park iD, Jeongwon Min iD, Sumin Park iD, Ju Yeon Kim iD, Jinsu Eun iD, Kyuha Jung iD, Yoobin Elyson Park iD, Esther Kim iD, Eun Young Lee iD, Joonhwan Lee iD, Jinwook Choi iD

JMIR Medical Informatics · 2024

DOI: 10.2196/52897 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Informatics, 2024. Verfasst von Kyungmo Kim und 11 weiteren Autor:innen. Beteiligte Einrichtung: Interdisciplinary Program for Bioengineering, Seoul National University, Seoul, Republic of Korea. Hauptthemen laut MEDLINE: Natural Language Processing, Multilingualism, Algorithms. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The bidirectional encoder representations from transformers (BERT) model has attracted considerable attention in clinical applications, such as patient classification and disease prediction. However, current studies have typically progressed to application development without a thorough assessment of the model’s comprehension of clinical context. Furthermore, limited comparative studies have been conducted on BERT models using medical documents from non–English-speaking countries. Therefore, the applicability of BERT models trained on English clinical notes to non-English contexts is yet to be confirmed. To address these gaps in literature, this study focused on identifying the most effective BERT model for non-English clinical notes.

Objective

In this study, we evaluated the contextual understanding abilities of various BERT models applied to mixed Korean and English clinical notes. The objective of this study was to identify the BERT model that excels in understanding the context of such documents.

Methods

Using data from 164,460 patients in a South Korean tertiary hospital, we pretrained BERT-base, BERT for Biomedical Text Mining (BioBERT), Korean BERT (KoBERT), and Multilingual BERT (M-BERT) to improve their contextual comprehension capabilities and subsequently compared their performances in 7 fine-tuning tasks.

Results

The model performance varied based on the task and token usage. First, BERT-base and BioBERT excelled in tasks using classification ([CLS]) token embeddings, such as document classification. BioBERT achieved the highest F 1 -score of 89.32. Both BERT-base and BioBERT demonstrated their effectiveness in document pattern recognition, even with limited Korean tokens in the dictionary. Second, M-BERT exhibited a superior performance in reading comprehension tasks, achieving an F 1 -score of 93.77. Better results were obtained when fewer words were replaced with unknown ([UNK]) tokens. Third, M-BERT excelled in the knowledge inference task in which correct disease names were inferred from 63 candidate disease names in a document with disease names replaced with [MASK] tokens. M-BERT achieved the highest hit@10 score of 95.41.

Conclusions

This study highlighted the effectiveness of various BERT models in a multilingual clinical domain. The findings can be used as a reference in clinical and language-based applications.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingMultilingualismAlgorithmsHumansRepublic of KoreaElectronic Health RecordsLanguageData Mining

Stichwörter der Autor:innen

NLPlarge language modelsnatural language inferencenatural language processingreading comprehensiontransformer

Autor:innen und Institutionen

  • Interdisciplinary Program for Bioengineering, Seoul National University, Seoul, Republic of Korea.
  • Seoul National University Medical Research Center, Seoul, Republic of Korea.
  • Institute of Medical and Biological Engineering, Medical Research Center, Seoul National University, Seoul, Republic of Korea.
  • Division of Rheumatology, Department of Internal Medicine, Seoul National University Hospital, Seoul, Republic of Korea.
  • Human Computer Interaction and Design Lab, Seoul National University, Seoul, Republic of Korea.
  • Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, Republic of Korea, 82 2-766-3421.

ORCID-Profile: Kyungmo Kim, Seongkeun Park, Jeongwon Min, Sumin Park, Ju Yeon Kim, Jinsu Eun, Kyuha Jung, Yoobin Elyson Park, Esther Kim, Eun Young Lee, Joonhwan Lee, Jinwook Choi

Zitieren

APA

Kim, K., Park, S., Min, J., Park, S., Kim, J. Y., Eun, J., Jung, K., Park, Y. E., Kim, E., Lee, E. Y., Lee, J., & Choi, J. (2024). Multifaceted Natural Language Processing Task–Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation. JMIR Medical Informatics. https://doi.org/10.2196/52897

Vancouver

Kim K, Park S, Min J, Park S, Kim JY, Eun J, et al. Multifaceted Natural Language Processing Task–Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation. JMIR Medical Informatics. 2024. doi:10.2196/52897

BibTeX
@article{Kim2024,
  author = {Kim, Kyungmo and Park, Seongkeun and Min, Jeongwon and Park, Sumin and Kim, Ju Yeon and Eun, Jinsu and Jung, Kyuha and Park, Yoobin Elyson and Kim, Esther and Lee, Eun Young and Lee, Joonhwan and Choi, Jinwook},
  title = {Multifaceted Natural Language Processing Task–Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation},
  journal = {JMIR Medical Informatics},
  year = {2024},
  doi = {10.2196/52897}
}
Kostenlos starten →