EUVIMEDOpdag EUVIMED
Open access

Multifaceted Natural Language Processing Task–Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation

Kyungmo Kim iD, Seongkeun Park iD, Jeongwon Min iD, Sumin Park iD, Ju Yeon Kim iD, Jinsu Eun iD, Kyuha Jung iD, Yoobin Elyson Park iD, Esther Kim iD, Eun Young Lee iD, Joonhwan Lee iD, Jinwook Choi iD

JMIR Medical Informatics · 2024

DOI: 10.2196/52897 ↗

At a glance

Journal article in JMIR Medical Informatics, 2024. Written by Kyungmo Kim and 11 co-authors. Institution: Interdisciplinary Program for Bioengineering, Seoul National University, Seoul, Republic of Korea. Major MEDLINE topics: Natural Language Processing, Multilingualism, Algorithms. Cited 9 times so far (Crossref); 44 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

The bidirectional encoder representations from transformers (BERT) model has attracted considerable attention in clinical applications, such as patient classification and disease prediction. However, current studies have typically progressed to application development without a thorough assessment of the model’s comprehension of clinical context. Furthermore, limited comparative studies have been conducted on BERT models using medical documents from non–English-speaking countries. Therefore, the applicability of BERT models trained on English clinical notes to non-English contexts is yet to be confirmed. To address these gaps in literature, this study focused on identifying the most effective BERT model for non-English clinical notes.

Objective

In this study, we evaluated the contextual understanding abilities of various BERT models applied to mixed Korean and English clinical notes. The objective of this study was to identify the BERT model that excels in understanding the context of such documents.

Methods

Using data from 164,460 patients in a South Korean tertiary hospital, we pretrained BERT-base, BERT for Biomedical Text Mining (BioBERT), Korean BERT (KoBERT), and Multilingual BERT (M-BERT) to improve their contextual comprehension capabilities and subsequently compared their performances in 7 fine-tuning tasks.

Results

The model performance varied based on the task and token usage. First, BERT-base and BioBERT excelled in tasks using classification ([CLS]) token embeddings, such as document classification. BioBERT achieved the highest F 1 -score of 89.32. Both BERT-base and BioBERT demonstrated their effectiveness in document pattern recognition, even with limited Korean tokens in the dictionary. Second, M-BERT exhibited a superior performance in reading comprehension tasks, achieving an F 1 -score of 93.77. Better results were obtained when fewer words were replaced with unknown ([UNK]) tokens. Third, M-BERT excelled in the knowledge inference task in which correct disease names were inferred from 63 candidate disease names in a document with disease names replaced with [MASK] tokens. M-BERT achieved the highest hit@10 score of 95.41.

Conclusions

This study highlighted the effectiveness of various BERT models in a multilingual clinical domain. The findings can be used as a reference in clinical and language-based applications.

Subjects

MeSH terms (MEDLINE)

Natural Language ProcessingMultilingualismAlgorithmsHumansRepublic of KoreaElectronic Health RecordsLanguageData Mining

Author keywords

NLPlarge language modelsnatural language inferencenatural language processingreading comprehensiontransformer

Authors and institutions

  • Interdisciplinary Program for Bioengineering, Seoul National University, Seoul, Republic of Korea.
  • Seoul National University Medical Research Center, Seoul, Republic of Korea.
  • Institute of Medical and Biological Engineering, Medical Research Center, Seoul National University, Seoul, Republic of Korea.
  • Division of Rheumatology, Department of Internal Medicine, Seoul National University Hospital, Seoul, Republic of Korea.
  • Human Computer Interaction and Design Lab, Seoul National University, Seoul, Republic of Korea.
  • Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, Republic of Korea, 82 2-766-3421.

ORCID profiles: Kyungmo Kim, Seongkeun Park, Jeongwon Min, Sumin Park, Ju Yeon Kim, Jinsu Eun, Kyuha Jung, Yoobin Elyson Park, Esther Kim, Eun Young Lee, Joonhwan Lee, Jinwook Choi

Cite

APA

Kim, K., Park, S., Min, J., Park, S., Kim, J. Y., Eun, J., Jung, K., Park, Y. E., Kim, E., Lee, E. Y., Lee, J., & Choi, J. (2024). Multifaceted Natural Language Processing Task–Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation. JMIR Medical Informatics. https://doi.org/10.2196/52897

Vancouver

Kim K, Park S, Min J, Park S, Kim JY, Eun J, et al. Multifaceted Natural Language Processing Task–Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation. JMIR Medical Informatics. 2024. doi:10.2196/52897

BibTeX
@article{Kim2024,
  author = {Kim, Kyungmo and Park, Seongkeun and Min, Jeongwon and Park, Sumin and Kim, Ju Yeon and Eun, Jinsu and Jung, Kyuha and Park, Yoobin Elyson and Kim, Esther and Lee, Eun Young and Lee, Joonhwan and Choi, Jinwook},
  title = {Multifaceted Natural Language Processing Task–Based Evaluation of Bidirectional Encoder Representations From Transformers Models for Bilingual (Korean and English) Clinical Notes: Algorithm Development and Validation},
  journal = {JMIR Medical Informatics},
  year = {2024},
  doi = {10.2196/52897}
}
Start for free →