From Agreement to Action: A Reliability Contract for Multi-Judge LLM Assurance
Proceedings of the AAAI Symposium Series · 06.11.2026 · 10 (1) · 433-437
DOI: 10.1609/aaaiss.v10i1.43436 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Symposium Series, online erschienen am 06.11.2026. Verfasst von Khrystyna Terletska.
Abstract (Originalsprache)
Multi-judge LLM evaluation often treats agreement as evidence of reliability. We test this assumption on 350 objectively labeled JudgeBench response pairs using five judges and both answer orders. Judge selection, weighting, and abstention thresholds are fitted on four folds and applied unchanged to the held-out fold. On 277 complete cases, five-judge majority voting achieves 76.9% accuracy, compared with 84.5% for the cross-fitted single judge. At an 80% coverage target, vote-margin abstention has 17.6% error, our reliability contract has 9.4%, and the single judge with a position-stability gate has 7.8%. Seven of 64 majority-vote errors occur under unanimity. Panel size and consensus are therefore not reliability guarantees. Safe automation requires validated judges, measured error dependence, selective action, and audits of unanimous decisions.
Zitieren
APA
Terletska, K. (2026). From Agreement to Action: A Reliability Contract for Multi-Judge LLM Assurance. Proceedings of the AAAI Symposium Series, 10(1), 433-437. https://doi.org/10.1609/aaaiss.v10i1.43436
Vancouver
Terletska K. From Agreement to Action: A Reliability Contract for Multi-Judge LLM Assurance. Proceedings of the AAAI Symposium Series. 2026;10(1):433-437. doi:10.1609/aaaiss.v10i1.43436
BibTeX
@article{Terletska2026,
author = {Terletska, Khrystyna},
title = {From Agreement to Action: A Reliability Contract for Multi-Judge LLM Assurance},
journal = {Proceedings of the AAAI Symposium Series},
year = {2026},
volume = {10},
number = {1},
pages = {433--437},
doi = {10.1609/aaaiss.v10i1.43436}
}
EUVIMED