Does Question Really Matter? The Attribution of Answer Bias in LLM Evaluation
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (36) · 30130-30138
DOI: 10.1609/aaai.v40i36.40262 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Boxi Cao und 4 weiteren Autor:innen.
Abstract (Originalsprache)
Multiple-choices question answering (MCQA) has emerged as one of the most popular task formats for large language models (LLMs) evaluation. Unfortunately, there exist substantial evidence that the evaluation of current MCQA benchmarks suffers from significant answer bias, which severely undermines the reliability of the evaluation conclusions. Specifically, many LLMs achieve performance significantly higher than random selection even when the questions are omitted from input information. To this end, we conduct a systematic investigation of the attribution of answer bias, and demonstrate a strong correlation between the degree of data contamination and the severity of answer bias, while the position of options and the popularity of answers have relatively minor effects. Building on these insights, we further propose OPD, a straightforward yet effective tool for contamination detection and dataset debiasing without requiring access to the model’s internal training data. Our findings and algorithms provide valuable insights for the design of future trustworthy LLM evaluation protocols.
Zitieren
APA
Cao, B., Pan, R., Lin, H., Han, X., & Sun, L. (2026). Does Question Really Matter? The Attribution of Answer Bias in LLM Evaluation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(36), 30130-30138. https://doi.org/10.1609/aaai.v40i36.40262
Vancouver
Cao B, Pan R, Lin H, Han X, Sun L. Does Question Really Matter? The Attribution of Answer Bias in LLM Evaluation. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(36):30130-30138. doi:10.1609/aaai.v40i36.40262
BibTeX
@article{Cao2026,
author = {Cao, Boxi and Pan, Ruotong and Lin, Hongyu and Han, Xianpei and Sun, Le},
title = {Does Question Really Matter? The Attribution of Answer Bias in LLM Evaluation},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {36},
pages = {30130--30138},
doi = {10.1609/aaai.v40i36.40262}
}
EUVIMED