EUVIMEDEUVIMED entdecken
ValidierungsstudieOpen Access

Automated Multitier Tagging of Chinese Online Health Education Resources Using a Large Language Model: Development and Validation Study

Jialin Meng iD, Ruiming Dai iD, Xiaolan Huang iD, Yi Gu iD, Shixing Yan iD, Xiaoke Wang iD, Jingrong Gao iD, Tian-Tian Zhang iD

Journal of Medical Internet Research · 2025

DOI: 10.2196/83219 ↗

Kurz gefasst

Validierungsstudie in der Zeitschrift Journal of Medical Internet Research, 2025. Verfasst von Jialin Meng und 7 weiteren Autor:innen. Beteiligte Einrichtung: School of Public Health, Fudan University, Shanghai, China. Hauptthemen laut MEDLINE: Health Education, Internet, Language. Die Arbeit nennt 36 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Precision health promotion, which aims to tailor health messages to individual needs, is hampered by the lack of structured metadata in vast digital health resource libraries. This bottleneck prevents scalable, personalized content delivery and exacerbates information overload for the public.

Objective

This study aimed to develop, deploy, and validate an automated tagging system using a large language model (LLM) to create the foundational metadata infrastructure required for tailored health communication at scale.

Methods

We developed a comprehensive, 3-tier health promotion taxonomy (10 primary, 34 secondary, and 90,562 tertiary tags) using a hybrid Delphi and corpus-mining methodology. We then constructed a hybrid inference pipeline by fine-tuning a Baichuan2-7B LLM with low-rank adaptation for initial tag generation. This was then refined by a domain-specific named entity recognition model and standardized against a vector database. The system’s performance was evaluated against manual annotations from nonexpert staff on a test set of 1000 resources. We used a “no gold standard” framework, comparing the artificial intelligence–human (A-H) interrater reliability (IRR) with a supplemental human-human (H-H) IRR baseline and expert adjudication for cases where artificial intelligence provided additional tags (“AI Additive”).

Results

The A-H agreement was moderate (Cohen κ=0.54, 95% CI 0.53-0.56; Jaccard similarity coefficient=0.48, 95% CI 0.46-0.50). Critically, this was higher than the baseline nonexpert H-H agreement (Cohen κ=0.32, 95% CI 0.29-0.35; Jaccard similarity coefficient=0.35, 95% CI 0.27-0.43). A granular analysis of disagreements revealed that in 15.9% (159/1000) of the cases, the “AI Additive” tags were not identified by human annotators. Expert adjudication of these cases confirmed that the “AI Additive” tags were correct and relevant with a precision of 90% (45/50; 95% CI 78.2%-96.7%).

Conclusions

A fine-tuned LLM, integrated into a hybrid pipeline, can function as a powerful augmentation tool for health content annotation. The system’s consistency (A-H κ=0.54) was found to be superior to the baseline human workflow (H-H κ=0.32). By moving beyond simple automation to reliably identify relevant health topics missed by manual annotators with high, expert-validated accuracy, this study provides a robust technical and methodological blueprint for implementing artificial intelligence to enhance precision health communication in public health settings.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Health EducationInternetLanguageHumansArtificial IntelligenceChinaLarge Language Models

Stichwörter der Autor:innen

Chinadigital healthhealth promotionlarge language modelnamed entity recognitionnatural language processingtagging

Autor:innen und Institutionen

  • School of Public Health, Fudan University, Shanghai, China.
  • Shanghai Center for Emerging Technologies Governance in Medicine and Public Health, Shanghai, China.
  • Shanghai Municipal Center for Health Promotion, Shanghai, China.

ORCID-Profile: Jialin Meng, Ruiming Dai, Xiaolan Huang, Yi Gu, Shixing Yan, Xiaoke Wang, Jingrong Gao, Tian-Tian Zhang

Zitieren

APA

Meng, J., Dai, R., Huang, X., Gu, Y., Yan, S., Wang, X., Gao, J., & Zhang, T. T. (2025). Automated Multitier Tagging of Chinese Online Health Education Resources Using a Large Language Model: Development and Validation Study. Journal of Medical Internet Research. https://doi.org/10.2196/83219

Vancouver

Meng J, Dai R, Huang X, Gu Y, Yan S, Wang X, et al. Automated Multitier Tagging of Chinese Online Health Education Resources Using a Large Language Model: Development and Validation Study. Journal of Medical Internet Research. 2025. doi:10.2196/83219

BibTeX
@article{Meng2025,
  author = {Meng, Jialin and Dai, Ruiming and Huang, Xiaolan and Gu, Yi and Yan, Shixing and Wang, Xiaoke and Gao, Jingrong and Zhang, Tian-Tian},
  title = {Automated Multitier Tagging of Chinese Online Health Education Resources Using a Large Language Model: Development and Validation Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/83219}
}
Kostenlos starten →