EUVIMEDEUVIMED entdecken
Open Access

Toward Cross-Hospital Deployment of Natural Language Processing Systems: Model Development and Validation of Fine-Tuned Large Language Models for Disease Name Recognition in Japanese

Seiji Shimizu iD, Tomohiro Nishiyama iD, Hiroyuki Nagai iD, Shoko Wakamiya iD, Eiji Aramaki iD

JMIR Medical Informatics · 2025

DOI: 10.2196/76773 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Medical Informatics, 2025. Verfasst von Seiji Shimizu und 4 weiteren Autor:innen. Beteiligte Einrichtung: Nara Institute of Science and Technology, 8916-5, Takayama-cho, Ikoma-shi, Nara, 630-0192, Japan, 81 743-72-5250. Hauptthemen laut MEDLINE: Disease, Electronic Health Records, Natural Language Processing. Die Arbeit wurde bisher 7-mal zitiert (Crossref) und nennt 28 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Disease name recognition is a fundamental task in clinical natural language processing, enabling the extraction of critical patient information from electronic health records. While recent advances in large language models (LLMs) have shown promise, most evaluations have focused on English, and little is known about their robustness in low-resource languages such as Japanese. In particular, whether these models can perform reliably on previously unseen in-hospital data, which differs from training data in writing styles and clinical contexts, has not been thoroughly investigated.

Objective

This study evaluated the robustness of fine-tuned LLMs for disease name recognition in Japanese clinical notes, with a particular focus on their performance on in-hospital data that was not included during training.

Methods

We used two corpora for this study: (1) a publicly available set of Japanese case reports denoted as CR, and (2) a newly constructed corpus of progress notes, denoted as PN, written by ten physicians to capture stylistic variations of in-hospital clinical notes. To reflect real-world deployment scenarios, we first fine-tuned models on CR. Specifically, we compared a LLM and a baseline-masked language model (MLM). These models were then evaluated under two conditions: (1) on CR, representing the in-domain (ID) setting with the same document type, similar to training, and (2) on PN, representing the out-of-domain (OOD) setting with a different document type. Robustness was assessed by calculating the performance gap (ie, the performance drop from in-domain to out-of-domain settings).

Results

The LLM demonstrated greater robustness, with a smaller performance gap in F 1 -scores (ID–OOD = −8.6) compared to the MLM baseline performance (ID–OOD = −13.9). This indicated more stable performance across ID and OOD settings, highlighting the effectiveness of fine-tuned LLMs for reliable use in diverse clinical settings.

Conclusions

Fine-tuned LLMs demonstrate superior robustness for disease name recognition in Japanese clinical notes, with a smaller performance gap. These findings highlight the potential of LLMs as reliable tools for clinical natural language processing in low-resource language settings and support their deployment in real-world health care applications, where diversity in documentation is inevitable.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

DiseaseElectronic Health RecordsNatural Language ProcessingHumansJapanLanguageLarge Language Models

Stichwörter der Autor:innen

Japanese languageclinical NLPclinical corpusclinical natural language processinglarge language modelsnamed entity recognitionout-of-domain robustness

Autor:innen und Institutionen

  • Nara Institute of Science and Technology, 8916-5, Takayama-cho, Ikoma-shi, Nara, 630-0192, Japan, 81 743-72-5250.

ORCID-Profile: Seiji Shimizu, Tomohiro Nishiyama, Hiroyuki Nagai, Shoko Wakamiya, Eiji Aramaki

Zitieren

APA

Shimizu, S., Nishiyama, T., Nagai, H., Wakamiya, S., & Aramaki, E. (2025). Toward Cross-Hospital Deployment of Natural Language Processing Systems: Model Development and Validation of Fine-Tuned Large Language Models for Disease Name Recognition in Japanese. JMIR Medical Informatics. https://doi.org/10.2196/76773

Vancouver

Shimizu S, Nishiyama T, Nagai H, Wakamiya S, Aramaki E. Toward Cross-Hospital Deployment of Natural Language Processing Systems: Model Development and Validation of Fine-Tuned Large Language Models for Disease Name Recognition in Japanese. JMIR Medical Informatics. 2025. doi:10.2196/76773

BibTeX
@article{Shimizu2025,
  author = {Shimizu, Seiji and Nishiyama, Tomohiro and Nagai, Hiroyuki and Wakamiya, Shoko and Aramaki, Eiji},
  title = {Toward Cross-Hospital Deployment of Natural Language Processing Systems: Model Development and Validation of Fine-Tuned Large Language Models for Disease Name Recognition in Japanese},
  journal = {JMIR Medical Informatics},
  year = {2025},
  doi = {10.2196/76773}
}
Kostenlos starten →