Human-AI Agreement in Citation Validation of an AI-Generated Subspecialty Anesthesiology Knowledge Base: Cross-Sectional Paired Validation Study
JMIR Metascience and Research Integrity · 19.10.2026 · 1 · e105947
DOI: 10.2196/105947 ↗Kurz gefasst
Querschnittstudie in der Zeitschrift JMIR Metascience and Research Integrity, online erschienen am 19.10.2026. Verfasst von Juan Li und 11 weiteren Autor:innen. Die Arbeit nennt 47 Literaturangaben.
Abstract (Originalsprache)
Background
Evaluating AI-generated citations requires checking article existence, clinical relevance, scientific accuracy, and translation fidelity. Agreement alone cannot explain judgments or their temporal stability. Retained records and repeat assessments can support transparent citation review in bilingual educational resources.
Objective
This study quantifies paired AI-human agreement at binary, 3-category, and criterion levels; evaluates AI alert rules against human FAIL ratings; compares original and repeat AI ratings; and examines evaluation records.
Methods
This cross-sectional study systematically sampled 570 OpenEvidence-generated citations across 19 subspecialties of a bilingual anesthesiology knowledge base. Each citation received 1 AI and 1 human overall verdict; 4 domains assessed existence, relevance, scientific accuracy, and translation fidelity. Ten validators represented 9 institutions; validator and subspecialty effects were inseparable. The primary post hoc binary comparison combined PASS and CONDITIONAL. Human verdicts served as a reference, not as truth. Records documented search and recovery. Applicable 95% CIs were calculated using tier-preserving subspecialty-cluster bootstrapping (B=10,000). AI reassessed 112 citations.
Results
Binary agreement was 489 out of 570 (85.8%; 95% CI 79.12%-91.75%); Cohen κ was 0.0645 (95% CI −0.0103 to 0.1639), Gwet first-order agreement coefficient was 0.8334 (95% CI 0.7388-0.9084), and Brennan-Prediger agreement was 0.7158 (95% CI 0.5825-0.8351). Three-category agreement was 271 out of 570 (47.5%; 95% CI 41.93%-53.16%). Agreement was lower for relevance (323/569, 56.8%; 95% CI 51.75%-61.62%) and scientific accuracy (440/568, 77.5%; 95% CI 72.63%-81.83%). Flagging AI CONDITIONAL/FAIL identified 274 out of 570 citations (48.1%; 95% CI 42.98%-52.98%) and captured 46 out of 74 human FAIL ratings (62%; 95% CI 48.27%-74.42%); the positive predictive value was 46 out of 274 (16.8%; 95% CI 7.89%-26.27%). Among unflagged citations, 28 out of 296 (9.5%) received human FAIL ratings (95% CI 4.84%-14.81%). In a later AI and retrieval run, unweighted agreement with original AI ratings was 91 out of 112 (81.3%) for 3 categories and 107 out of 112 (95.5%) for 2 categories. Original AI FAIL ratings persisted for 12 out of 17 citations overall and 2 out of 4 citations for article existence. According to human records, 389 citations were assigned no issue by rule without individual rereview; the principal investigator individually reviewed the remaining 181. Post hoc classification assigned 154 out of 570 (27%) citations to primary issue categories 1-5, including 33 out of 154 verification gaps; this classification is not an estimate of error prevalence.
Conclusions
This study combines paired AI-human judgments, citation checks, repeat AI assessment, and human review within a bilingual knowledge base. The framework highlights concerns requiring specialist review of source support, summaries, and translations. Retained records make citation evaluation inspectable, while public structured ratings enable statistical recalculation and reuse, supporting metascience and research integrity as AI-assisted evidence use expands.
Zitieren
APA
Li, J., Huang, S., Qiu, Y., Chen, W., Zhang, Y., Zhao, X., Qu, G., Bu, Y., Feng, X., Xu, J., Ma, H., & Wang, J. (2026). Human-AI Agreement in Citation Validation of an AI-Generated Subspecialty Anesthesiology Knowledge Base: Cross-Sectional Paired Validation Study. JMIR Metascience and Research Integrity, 1, e105947. https://doi.org/10.2196/105947
Vancouver
Li J, Huang S, Qiu Y, Chen W, Zhang Y, Zhao X, et al. Human-AI Agreement in Citation Validation of an AI-Generated Subspecialty Anesthesiology Knowledge Base: Cross-Sectional Paired Validation Study. JMIR Metascience and Research Integrity. 2026;1:e105947. doi:10.2196/105947
BibTeX
@article{Li2026,
author = {Li, Juan and Huang, Shun and Qiu, Yue and Chen, Wei and Zhang, Yang and Zhao, Xuechun and Qu, Ge and Bu, Yifan and Feng, Xiaomei and Xu, Jiawei and Ma, Haobo and Wang, Jingping},
title = {Human-AI Agreement in Citation Validation of an AI-Generated Subspecialty Anesthesiology Knowledge Base: Cross-Sectional Paired Validation Study},
journal = {JMIR Metascience and Research Integrity},
year = {2026},
volume = {1},
pages = {e105947},
doi = {10.2196/105947}
}
EUVIMED