Context-Aware Sentence Classification of Radiology Reports Using Synthetic Data: Development and Validation Study
Journal of Medical Internet Research · 2026
DOI: 10.2196/86365 ↗Kurz gefasst
Validierungsstudie in der Zeitschrift Journal of Medical Internet Research, 2026. Verfasst von Tomohiro Kikuchi und 6 weiteren Autor:innen. Beteiligte Einrichtung: Data Science Center, Jichi Medical University, Tochigi, Japan. Hauptthemen laut MEDLINE: Natural Language Processing, Radiology. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 47 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Automated structuring of radiology reports is essential for data utilization and the development of medical artificial intelligence models. However, manual annotation by experts is labor-intensive, and processing real clinical data through commercial large language models (LLMs) presents significant privacy risks. These challenges are particularly pronounced for non-English languages like Japanese, where specialized medical corpora are scarce. While synthetic data generation offers a potential privacy-preserving alternative, its effectiveness in capturing complex clinical nuances—such as negation and contextual dependencies—to train robust classification models without any real-world training data has not been fully established.
Objective
This study aimed to develop a context-aware sentence classification model for Japanese radiology reports using an entirely synthetic training pipeline, thereby eliminating reliance on real-world clinical data during the development phase. Furthermore, we sought to evaluate the generalizability of this approach by validating the model’s performance on diverse, multi-institutional, real-world reports.
Methods
Japanese radiology reports (n=3104) were generated using GPT-4.1 and automatically annotated at the sentence level into 4 categories (background, positive finding, negative finding, and continuation) using GPT-4.1-mini. The synthetic data were partitioned into training (n=2670), validation (n=334), and test (n=100) sets. We fine-tuned several models, including lightweight local LLMs (Qwen3 and Llama 3.2 series) using low-rank adaptation and Japanese text classification models (Bidirectional Encoder Representations from Transformers [BERT]-base Japanese v3, Japanese Medical Robustly Optimized BERT Pretraining Approach [JMedRoBERTa]-base, and ModernBERT-Ja-130M). External validation was performed using 280 real-world reports (3477 sentences) from 7 institutions in the Japan Medical Image Database, with ground-truth labels established by board-certified radiologists. Evaluation metrics included accuracy, macro-averaged F 1 (macro F 1 ) score, and positive predictive value for positive findings (PPV_1).
Results
All models achieved high performance on the synthetic test set (accuracy: 0.938‐0.951; macro F 1 -score: 0.924‐0.940). Overall performance declined on the external validation dataset (accuracy: 0.783‐0.813; macro F 1 -score: 0.761‐0.790), reflecting distributional differences between synthetic and real-world reports; however, PPV_1 remained stable and high across datasets (eg, 0.957 on the synthetic test set vs 0.952 on the external validation dataset for Qwen3 [4B]). Parsing errors occurred in LLM-based approaches (19‐260 sentences, 0.55%‐7.48% in the external dataset).
Conclusions
This study demonstrates the feasibility of developing context-aware sentence classification models for Japanese radiology reports using a training pipeline based entirely on synthetic data. The stability of PPV_1 indicates that the models successfully captured the essential clinical terminology and linguistic patterns required to identify positive findings in real-world reports, despite the observed performance degradation during external validation. This approach substantially reduces manual annotation requirements and privacy risks, providing a scalable foundation for constructing structured radiology datasets to support the development of clinically relevant medical artificial intelligence models.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Data Science Center, Jichi Medical University, Tochigi, Japan.
- Department of Radiology, Jichi Medical University, 3311-1, Yakushiji, Shimotsuke, Tochigi, 329-0498, Japan, 81 285-58-7362.
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Tokyo, Japan.
- Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan.
- Department of Radiology, Juntendo University School of Medicine, Tokyo, Japan.
ORCID-Profile: Tomohiro Kikuchi, Yosuke Yamagishi, Kohei Yamamoto, Toshiaki Akashi, Harushi Mori, Hisaki Makimoto, Takahide Kohro
Zitieren
APA
Kikuchi, T., Yamagishi, Y., Yamamoto, K., Akashi, T., Mori, H., Makimoto, H., & Kohro, T. (2026). Context-Aware Sentence Classification of Radiology Reports Using Synthetic Data: Development and Validation Study. Journal of Medical Internet Research. https://doi.org/10.2196/86365
Vancouver
Kikuchi T, Yamagishi Y, Yamamoto K, Akashi T, Mori H, Makimoto H, et al. Context-Aware Sentence Classification of Radiology Reports Using Synthetic Data: Development and Validation Study. Journal of Medical Internet Research. 2026. doi:10.2196/86365
BibTeX
@article{Kikuchi2026,
author = {Kikuchi, Tomohiro and Yamagishi, Yosuke and Yamamoto, Kohei and Akashi, Toshiaki and Mori, Harushi and Makimoto, Hisaki and Kohro, Takahide},
title = {Context-Aware Sentence Classification of Radiology Reports Using Synthetic Data: Development and Validation Study},
journal = {Journal of Medical Internet Research},
year = {2026},
doi = {10.2196/86365}
}
EUVIMED