CRUCIBLE: A Multi-Agent Framework for Automated Annotation Rubric Calibration via Iterative Inter-Judge Disagreement Resolution
Proceedings of the AAAI Symposium Series · 06.11.2026 · 10 (1) · 275-283
DOI: 10.1609/aaaiss.v10i1.43416 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Symposium Series, online erschienen am 06.11.2026. Verfasst von Bidit Das.
Abstract (Originalsprache)
Annotation quality is a limiting factor in supervised learning pipelines that rely on the LLM-as-judge paradigm. This paper's central finding is that an automated convergence criterion can certify agreement that chance-corrected measurement does not support, and that the certification is an artifact of the agreement metric being optimized. Our weighted agreement statistic has a floor of 0.667 at zero unanimity, so a convergence threshold of τ = 0.80 is satisfied at 8 of 20 unanimous items: the loop halts and declares success while 60% of items still carry judge disagreement. The domain that crossed τ on this metric at 81.7% has a chance-corrected inter-judge agreement of only AC1 = 0.286. We reach this finding while building CRUCIBLE, a multi-agent system that automatically calibrates annotation rubrics through iterative, coordinator-mediated disagreement diagnosis. No human is in the loop after initialization, and no human reference labels are used during calibration. Applied to 60 YouTube transcripts across three domains, CRUCIBLE reached 96.7%, 81.7%, and 70.0% weighted inter-judge agreement, corresponding to chance-corrected AC1 of 0.929, 0.286, and -0.080. An independent human validation study covering the full corpus—20 transcripts per domain, scored by two disjoint cohorts of three annotators each—finds that judge and human chance-corrected reliability rank the three domains identically, so the judges find difficult what humans find difficult. We report as a negative result that agreement between human consensus and the system's labels carries almost no evidential weight in this design: five of six domain-round cells fail to beat a majority-class baseline that labels every item with the domain's dominant class, and the pooled rate of 75% is below the corresponding baseline. Against that baseline, a between-domain contrast we previously reported as preliminary evidence of a human-alignment dissociation does not survive. Two further interventions—porting the calibrated rubric to a frontier judge, and adding a reasoning step before the label—each move the optimized metric without moving agreement with the human reference, and a reasoning step alone moves it by +11.6 points. These results locate CRUCIBLE's value not in raising raw agreement, and not in match against human consensus, but in diagnosing whether residual disagreement is rubric-fixable, capability-bounded, or prevalence-dominated—and in the accompanying warning that a stopping rule defined on an uncorrected agreement metric will terminate before agreement has been achieved. For a safety-critical evaluation pipeline, this is the risk-relevant contribution: an operator reading 81.7% agreement as convergence is reading a number whose floor is 66.7%.
Zitieren
APA
Das, B. (2026). CRUCIBLE: A Multi-Agent Framework for Automated Annotation Rubric Calibration via Iterative Inter-Judge Disagreement Resolution. Proceedings of the AAAI Symposium Series, 10(1), 275-283. https://doi.org/10.1609/aaaiss.v10i1.43416
Vancouver
Das B. CRUCIBLE: A Multi-Agent Framework for Automated Annotation Rubric Calibration via Iterative Inter-Judge Disagreement Resolution. Proceedings of the AAAI Symposium Series. 2026;10(1):275-283. doi:10.1609/aaaiss.v10i1.43416
BibTeX
@article{Das2026,
author = {Das, Bidit},
title = {CRUCIBLE: A Multi-Agent Framework for Automated Annotation Rubric Calibration via Iterative Inter-Judge Disagreement Resolution},
journal = {Proceedings of the AAAI Symposium Series},
year = {2026},
volume = {10},
number = {1},
pages = {275--283},
doi = {10.1609/aaaiss.v10i1.43416}
}
EUVIMED