EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

Fine-Tuning Large Language Models for Structured Extraction of Infectious Disease–Related Information From Clinical Notes in Japanese Primary Care: Development and Internal Validation Study

Hiroshi Yoshihara, Haruka Maeda, Yuriko Hagiwara, Daichi Sato, Kei Kitajima, Akihiro Iwata, Nicolas Van de Velde, Yuta Nakamura, Yosuke Yamagishi, Ataru Igarashi

JMIR Formative Research · 10.09.2026 · 10 · e84974-e84974

DOI: 10.2196/84974 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift JMIR Formative Research, online erschienen am 10.09.2026. Verfasst von Hiroshi Yoshihara und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Health Policy and Public Health, Graduate School of Pharmaceutical Sciences, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 11…. Hauptthemen laut MEDLINE: Primary Health Care, Electronic Health Records, Natural Language Processing, Communicable Diseases. Die Arbeit nennt 33 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The COVID-19 pandemic highlighted the importance of timely infectious disease surveillance. In Japan, conventional sentinel and claims-based systems incur reporting lags and capture limited clinical detail, whereas free-text clinical notes in electronic health records (EHRs) hold richer, timelier symptom and vaccination information. Natural language processing (NLP) with large language models (LLMs) offers a way to structure such free text at scale.

Objective

We aimed to develop and internally validate an NLP algorithm to extract structured infectious disease–related symptoms and vaccination history from free-text clinical notes in Japanese primary care, as a feasibility step toward low-latency, EHR-based surveillance.

Methods

A total of 773 clinical notes, originating from 526 unique patients, were provided by M3 Inc through the Japan Medical Data Survey and used for analysis. Three physicians annotated information related to infectious disease symptoms and vaccination history. The data were divided into 622 (80%) training cases and 151 (20%) evaluation cases with no patient overlap. We compared a physician-designed, rule-based algorithm, few-shot learning (FSL) using commercial and open-source LLMs, and supervised fine-tuning (SFT) of open-source LLMs, using the macroaveraged F 1 -score (unweighted mean across 9 clinical categories). Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were also computed, with 95% CIs from a patient-level cluster bootstrap (2000 replicates).

Results

Rule-based extraction achieved a macroaveraged F 1 -score of 0.685 (95% CI 0.630-0.736). FSL markedly improved the extraction of high-variability items such as vaccination history and onset date. Anthropic Claude 3.5 Sonnet achieved a macroaveraged F 1 -score of 0.875 (95% CI 0.800-0.913; sensitivity 0.929, specificity 0.918). SFT of Google’s open-source Gemma 2 27B model with quantized low-rank adaptation (QLoRA) achieved the highest point estimate (macroaveraged F 1 -score of 0.906, 95% CI 0.833-0.945; sensitivity 0.921, specificity 0.969, PPV 0.906); the difference from Claude 3.5 Sonnet was small and not statistically distinguishable (Δ F 1 -score=0.030, 95% CI −0.035 to 0.140). A small, fine-tuned Gemma 2 2B model reached 0.822 (95% CI 0.752-0.874), significantly lower than that of the 27B model (Δ F 1 -score=0.084, 95% CI 0.040-0.163).

Conclusions

A fine-tuned open-source LLM can accurately extract and structure infectious disease–related information from Japanese free-text clinical notes, achieving performance comparable to that of a commercial model while enabling processing within a closed environment. These findings support the feasibility of EHR-based digital surveillance, whose downstream utility remains to be demonstrated.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Primary Health CareElectronic Health RecordsNatural Language ProcessingCommunicable DiseasesHumansJapanLarge Language ModelsMaleFemaleAlgorithmsMiddle AgedCOVID-19East Asian People

Stichwörter der Autor:innen

EHRelectronic health recordinfectious diseaseslarge language modelnatural language processingpublic health

Autor:innen und Institutionen

  • Department of Health Policy and Public Health, Graduate School of Pharmaceutical Sciences, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-0033, Japan, 81 3-5841-4828.
  • Department of Respiratory Infections, Institute of Tropical Medicine, Nagasaki University, Nagasaki, Japan.
  • M3 Inc., Tokyo, Japan.
  • Moderna, Inc., Cambridge, MA, United States.
  • Department of Computational Diagnostic Radiology and Preventive Medicine, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan.
  • Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan.

Zitieren

APA

Yoshihara, H., Maeda, H., Hagiwara, Y., Sato, D., Kitajima, K., Iwata, A., Van de Velde, N., Nakamura, Y., Yamagishi, Y., & Igarashi, A. (2026). Fine-Tuning Large Language Models for Structured Extraction of Infectious Disease–Related Information From Clinical Notes in Japanese Primary Care: Development and Internal Validation Study. JMIR Formative Research, 10, e84974-e84974. https://doi.org/10.2196/84974

Vancouver

Yoshihara H, Maeda H, Hagiwara Y, Sato D, Kitajima K, Iwata A, et al. Fine-Tuning Large Language Models for Structured Extraction of Infectious Disease–Related Information From Clinical Notes in Japanese Primary Care: Development and Internal Validation Study. JMIR Formative Research. 2026;10:e84974-e84974. doi:10.2196/84974

BibTeX
@article{Yoshihara2026,
  author = {Yoshihara, Hiroshi and Maeda, Haruka and Hagiwara, Yuriko and Sato, Daichi and Kitajima, Kei and Iwata, Akihiro and Van de Velde, Nicolas and Nakamura, Yuta and Yamagishi, Yosuke and Igarashi, Ataru},
  title = {Fine-Tuning Large Language Models for Structured Extraction of Infectious Disease–Related Information From Clinical Notes in Japanese Primary Care: Development and Internal Validation Study},
  journal = {JMIR Formative Research},
  year = {2026},
  volume = {10},
  pages = {e84974--e84974},
  doi = {10.2196/84974}
}
Kostenlos starten →