EUVIMEDEUVIMED entdecken
Crossref

When Debate Fails: An Empirical Study of Incentive Misalignment in Prover–Estimator Games

Hannah Guan, Charles Hou

Proceedings of the AAAI Symposium Series · 23.06.2026 · 9 (1) · 43-51

DOI: 10.1609/aaaiss.v9i1.42904 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Symposium Series, online erschienen am 23.06.2026. Verfasst von Hannah Guan und 1 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Debate has been proposed as a mechanism for eliciting truthful reasoning from powerful learning agents by structuring interaction as a game between competing provers evaluated by an estimator. While theoretically appealing, the practical effectiveness of such incentive-based mechanisms remains underexplored. We present the first end-to-end empirical instantiation of this protocol using reinforcement learning agents with large language model on verifiable long-context reasoning tasks. Despite careful alignment of rewards with theoretical assumptions, we find that the debate mechanism consistently fails to elicit truthful or robust reasoning. Instead, agents converge to degenerate strategies that exploit estimator weaknesses or collapse into non-informative equilibria. Through controlled experiments and representation analysis using PCA and t-SNE, we identify a fundamental incentive–optimization mismatch: equilibrium incentives do not reliably translate into learnable equilibria under gradient-based optimization. Our results highlight a gap between game-theoretic guarantees and learned agent behavior, raising concerns about the reliability of debate-based alignment mechanisms in practice. We conclude with implications for incentive design and future directions for empirically grounded mechanism design in multi-agent systems.

Zitieren

APA

Guan, H., & Hou, C. (2026). When Debate Fails: An Empirical Study of Incentive Misalignment in Prover–Estimator Games. Proceedings of the AAAI Symposium Series, 9(1), 43-51. https://doi.org/10.1609/aaaiss.v9i1.42904

Vancouver

Guan H, Hou C. When Debate Fails: An Empirical Study of Incentive Misalignment in Prover–Estimator Games. Proceedings of the AAAI Symposium Series. 2026;9(1):43-51. doi:10.1609/aaaiss.v9i1.42904

BibTeX
@article{Guan2026,
  author = {Guan, Hannah and Hou, Charles},
  title = {When Debate Fails: An Empirical Study of Incentive Misalignment in Prover–Estimator Games},
  journal = {Proceedings of the AAAI Symposium Series},
  year = {2026},
  volume = {9},
  number = {1},
  pages = {43--51},
  doi = {10.1609/aaaiss.v9i1.42904}
}
Kostenlos starten →