EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

Context-Aware Sentence Classification of Radiology Reports Using Synthetic Data: Development and Validation Study

Tomohiro Kikuchi iD, Yosuke Yamagishi iD, Kohei Yamamoto iD, Toshiaki Akashi iD, Harushi Mori iD, Hisaki Makimoto iD, Takahide Kohro iD

Journal of Medical Internet Research · 2026

DOI: 10.2196/86365 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Journal of Medical Internet Research, 2026. Verfasst von Tomohiro Kikuchi und 6 weiteren Autor:innen. Beteiligte Einrichtung: Data Science Center, Jichi Medical University, Tochigi, Japan. Hauptthemen laut MEDLINE: Natural Language Processing, Radiology. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Automated structuring of radiology reports is essential for data utilization and the development of medical artificial intelligence models. However, manual annotation by experts is labor-intensive, and processing real clinical data through commercial large language models (LLMs) presents significant privacy risks. These challenges are particularly pronounced for non-English languages like Japanese, where specialized medical corpora are scarce. While synthetic data generation offers a potential privacy-preserving alternative, its effectiveness in capturing complex clinical nuances—such as negation and contextual dependencies—to train robust classification models without any real-world training data has not been fully established.

Objective

This study aimed to develop a context-aware sentence classification model for Japanese radiology reports using an entirely synthetic training pipeline, thereby eliminating reliance on real-world clinical data during the development phase. Furthermore, we sought to evaluate the generalizability of this approach by validating the model’s performance on diverse, multi-institutional, real-world reports.

Methods

Japanese radiology reports (n=3104) were generated using GPT-4.1 and automatically annotated at the sentence level into 4 categories (background, positive finding, negative finding, and continuation) using GPT-4.1-mini. The synthetic data were partitioned into training (n=2670), validation (n=334), and test (n=100) sets. We fine-tuned several models, including lightweight local LLMs (Qwen3 and Llama 3.2 series) using low-rank adaptation and Japanese text classification models (Bidirectional Encoder Representations from Transformers [BERT]-base Japanese v3, Japanese Medical Robustly Optimized BERT Pretraining Approach [JMedRoBERTa]-base, and ModernBERT-Ja-130M). External validation was performed using 280 real-world reports (3477 sentences) from 7 institutions in the Japan Medical Image Database, with ground-truth labels established by board-certified radiologists. Evaluation metrics included accuracy, macro-averaged F 1 (macro F 1 ) score, and positive predictive value for positive findings (PPV_1).

Results

All models achieved high performance on the synthetic test set (accuracy: 0.938‐0.951; macro F 1 -score: 0.924‐0.940). Overall performance declined on the external validation dataset (accuracy: 0.783‐0.813; macro F 1 -score: 0.761‐0.790), reflecting distributional differences between synthetic and real-world reports; however, PPV_1 remained stable and high across datasets (eg, 0.957 on the synthetic test set vs 0.952 on the external validation dataset for Qwen3 [4B]). Parsing errors occurred in LLM-based approaches (19‐260 sentences, 0.55%‐7.48% in the external dataset).

Conclusions

This study demonstrates the feasibility of developing context-aware sentence classification models for Japanese radiology reports using a training pipeline based entirely on synthetic data. The stability of PPV_1 indicates that the models successfully captured the essential clinical terminology and linguistic patterns required to identify positive findings in real-world reports, despite the observed performance degradation during external validation. This approach substantially reduces manual annotation requirements and privacy risks, providing a scalable foundation for constructing structured radiology datasets to support the development of clinically relevant medical artificial intelligence models.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Natural Language ProcessingRadiologyLarge Language ModelsJapanHumansArtificial IntelligenceClassification Algorithms

Stichwörter der Autor:innen

artificial intelligencedata annotationlarge language modelsnatural language processingradiology report

Autor:innen und Institutionen

  • Data Science Center, Jichi Medical University, Tochigi, Japan.
  • Department of Radiology, Jichi Medical University, 3311-1, Yakushiji, Shimotsuke, Tochigi, 329-0498, Japan, 81 285-58-7362.
  • Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Tokyo, Japan.
  • Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan.
  • Department of Radiology, Juntendo University School of Medicine, Tokyo, Japan.

ORCID-Profile: Tomohiro Kikuchi, Yosuke Yamagishi, Kohei Yamamoto, Toshiaki Akashi, Harushi Mori, Hisaki Makimoto, Takahide Kohro

Zitieren

APA

Kikuchi, T., Yamagishi, Y., Yamamoto, K., Akashi, T., Mori, H., Makimoto, H., & Kohro, T. (2026). Context-Aware Sentence Classification of Radiology Reports Using Synthetic Data: Development and Validation Study. Journal of Medical Internet Research. https://doi.org/10.2196/86365

Vancouver

Kikuchi T, Yamagishi Y, Yamamoto K, Akashi T, Mori H, Makimoto H, et al. Context-Aware Sentence Classification of Radiology Reports Using Synthetic Data: Development and Validation Study. Journal of Medical Internet Research. 2026. doi:10.2196/86365

BibTeX
@article{Kikuchi2026,
  author = {Kikuchi, Tomohiro and Yamagishi, Yosuke and Yamamoto, Kohei and Akashi, Toshiaki and Mori, Harushi and Makimoto, Hisaki and Kohro, Takahide},
  title = {Context-Aware Sentence Classification of Radiology Reports Using Synthetic Data: Development and Validation Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  doi = {10.2196/86365}
}
Kostenlos starten →