EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC 4.0

Development of a robust corpus for automated evaluation of online health information in Chinese using the DISCERN scale

Ting E, Xingxi Li, Jun Liang, Junhao Ma, Qichuan Fang, Shanli Chen, Jianbo Lei, Christopher G Chute iD

Journal of the American Medical Informatics Association · 01.02.2026 · 33 (2) · 316-325

DOI: 10.1093/jamia/ocaf175 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 12.11.2025. Verfasst von Ting E und 7 weiteren Autor:innen. Beteiligte Einrichtung: Bloomberg School of Public Health,Johns Hopkins University, MD, 21205, United States. Hauptthemen laut MEDLINE: Internet, Consumer Health Information, Data Curation. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: Natural Science Foundation of Beijing.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To develop the first comprehensive, standardized annotated corpus of Chinese online health information (OHI) using the full 16-item DISCERN instrument and to establish a reliable annotation process that supports automated quality assessment. Materials and Methods We assembled 510 web-sourced articles on breast cancer, arthritis, and depression. All the articles were independently annotated by three trained raters using the DISCERN scale. Annotation followed a four-step workflow: data collection and preprocessing, rater training, iterative annotation, and quality control. Raters calibrated through consensus sessions and calibration articles. The Dawid–Skene model aggregated individual annotations into final consensus scores. Original five-point ratings were retained and binarized (scores 1-3 as low quality, 4-5 as high quality) to enable both fine-grained and coarse evaluation for machine learning.

Results

Initial annotation of a 60-article pilot produced low agreement (mean Krippendorff’s α ≈ 0.022) due to subjective variability. Successive calibration exercises improved agreement markedly, culminating in a corpus-wide Krippendorff’s α of 0.834. Consensus scores correlated strongly with individual rater scores, confirming annotation robustness. The dual-scale design yielded a relatively balanced distribution of labels across topics, with roughly equal representation of low- and high-quality articles, and preserved granularity for detailed DISCERN analysis.

Discussion

Our iterative calibration approach and consensus modeling effectively addressed the subjective ambiguity inherent in quality assessment. The binary and five-class labeling strategies facilitate flexible downstream applications, allowing automated systems to perform both broad filtering and nuanced quality differentiation. The high inter-rater reliability demonstrates that rigorous training and consensus methods can overcome domain-specific annotation challenges.

Conclusion

The resulting Chinese OHI corpus, annotated via a standardized DISCERN framework and refined through iterative calibration, provides a robust benchmark for training and evaluating machine learning models. This resource lays the foundation for scalable, reliable automated quality assessment of OHI in Chinese public health settings.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

InternetConsumer Health InformationData CurationHumansChinaMachine LearningBreast NeoplasmsDepression

Stichwörter der Autor:innen

Chinese online health informationDISCERN scaleannotated corpusautomated evaluationinter-rater reliabilitymachine learning

Autor:innen und Institutionen

  • Bloomberg School of Public Health,Johns Hopkins University, MD, 21205, United States.
  • Department of Industrial Engineering, Tsinghua University, Beijing, 100084, China.
  • Department of AI and IT, Second Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang Province, 310000, China.
  • School of Public Health, Hangzhou Medical College, Hangzhou, Zhejiang Province, 310053, China.
  • School of Medical Technology and Information Engineering, Zhejiang Chinese Medical University, Hangzhou, Zhejiang Province, 310053, China.
  • School of Public Health, Southwest Medical University, Luzhou, Sichuan Province, 646000, China.

ORCID-Profile: Christopher G Chute

Förderung

  • Natural Science Foundation of Beijing

Zitieren

APA

E, T., Li, X., Liang, J., Ma, J., Fang, Q., Chen, S., Lei, J., & Chute, C. G. (2026). Development of a robust corpus for automated evaluation of online health information in Chinese using the DISCERN scale. Journal of the American Medical Informatics Association, 33(2), 316-325. https://doi.org/10.1093/jamia/ocaf175

Vancouver

E T, Li X, Liang J, Ma J, Fang Q, Chen S, et al. Development of a robust corpus for automated evaluation of online health information in Chinese using the DISCERN scale. Journal of the American Medical Informatics Association. 2026;33(2):316-325. doi:10.1093/jamia/ocaf175

BibTeX
@article{E2026,
  author = {E, Ting and Li, Xingxi and Liang, Jun and Ma, Junhao and Fang, Qichuan and Chen, Shanli and Lei, Jianbo and Chute, Christopher G},
  title = {Development of a robust corpus for automated evaluation of online health information in Chinese using the DISCERN scale},
  journal = {Journal of the American Medical Informatics Association},
  year = {2026},
  volume = {33},
  number = {2},
  pages = {316--325},
  doi = {10.1093/jamia/ocaf175}
}
Kostenlos starten →