EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Benchmarking Choice Consistency in Large Language Models: A Measurement-Theoretic Analysis of Transitivity Axiom

Kiwon Song, James M. Jennings, Clintin P. Stober

Computational Brain & Behavior · 17.11.2026

DOI: 10.1007/s42113-026-00342-3 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computational Brain & Behavior, online erschienen am 17.11.2026. Verfasst von Kiwon Song und 2 weiteren Autor:innen. Die Arbeit nennt 72 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: U.S. Department of Defense, National Institutes of Health.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

While large language models (LLMs) are increasingly deployed as decision-makers, current evaluation practice emphasizes benchmark accuracy rather than the structural properties that make decisions coherent and interpretable. We introduce a measurement-theoretic framework for evaluating whether the choices of an LLM are transitively rationalizable, focusing on the choice axiom of transitivity of preference. When transitivity is violated over a choice set, alternatives can no longer be coherently ordered as globally better or worse, and a stated preference between one pair constrains nothing about how the model would choose between any others. We apply our framework across 20 distinct LLMs, systematically varying question format, generation temperature, and contextual memory, using classic choice paradigms previously shown to induce intransitive preferences in humans. We find substantial and systematic transitivity violations in many models, although some patterns partially mirror known features of human decision behavior. Testing the transitivity axiom thus offers a principled, complementary foundation for evaluating the consistency of AI decision making beyond benchmark accuracy.

Förderung

  • U.S. Department of Defense
  • National Institutes of Health

Zitieren

APA

Song, K., Jennings, J. M., & Stober, C. P. (2026). Benchmarking Choice Consistency in Large Language Models: A Measurement-Theoretic Analysis of Transitivity Axiom. Computational Brain & Behavior. https://doi.org/10.1007/s42113-026-00342-3

Vancouver

Song K, Jennings JM, Stober CP. Benchmarking Choice Consistency in Large Language Models: A Measurement-Theoretic Analysis of Transitivity Axiom. Computational Brain & Behavior. 2026. doi:10.1007/s42113-026-00342-3

BibTeX
@article{Song2026,
  author = {Song, Kiwon and Jennings, James M. and Stober, Clintin P.},
  title = {Benchmarking Choice Consistency in Large Language Models: A Measurement-Theoretic Analysis of Transitivity Axiom},
  journal = {Computational Brain & Behavior},
  year = {2026},
  doi = {10.1007/s42113-026-00342-3}
}
Kostenlos starten →