When Alignment Erases Disagreement: Preference Collapse and a Four-Valued Alternative for LLMs
Proceedings of the AAAI Symposium Series · 06.11.2026 · 10 (1) · 2-9
DOI: 10.1609/aaaiss.v10i1.43384 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Symposium Series, online erschienen am 06.11.2026. Verfasst von Aizierjiang Aiersilan.
Abstract (Originalsprache)
Alignment pipelines for large language models convert human judgments into a scalar reward and optimize a single policy against it. We argue that this conversion, not merely the demography of the annotator pool, makes contemporary alignment structurally insensitive to Global South, Indigenous, and other marginalized value frameworks. We present two results. First, under the Bradley--Terry model that underwrites RLHF and DPO, preference labels collected without annotator provenance from a deeply divided population and from a uniformly ambivalent one have identical distributions, so no estimator can distinguish a contested question from one about which users are indifferent. Second, the evaluation space that separates support, opposition, ignorance, and conflict is a three-dimensional simplex whose vertices are the four values of Belnap--Dunn logic; a scalar reward is the output of a map with two-dimensional fibers, which discards conflict entirely and folds ignorance evenly into support and opposition. Consensus-seeking elicitation does not repair the collapse; it removes contested items by construction. We therefore propose a four-valued architecture that returns admissible sets rather than maximizers, maintains a non-negotiable safety floor in the classical fragment while treating contested values paraconsistently, and exposes contestation to institutional governance instead of resolving it silently; we analyze its attack surface, use Sen's liberal paradox as a diagnostic map of the design space, and propose conflict-preserving evaluation metrics.
Zitieren
APA
Aiersilan, A. (2026). When Alignment Erases Disagreement: Preference Collapse and a Four-Valued Alternative for LLMs. Proceedings of the AAAI Symposium Series, 10(1), 2-9. https://doi.org/10.1609/aaaiss.v10i1.43384
Vancouver
Aiersilan A. When Alignment Erases Disagreement: Preference Collapse and a Four-Valued Alternative for LLMs. Proceedings of the AAAI Symposium Series. 2026;10(1):2-9. doi:10.1609/aaaiss.v10i1.43384
BibTeX
@article{Aiersilan2026,
author = {Aiersilan, Aizierjiang},
title = {When Alignment Erases Disagreement: Preference Collapse and a Four-Valued Alternative for LLMs},
journal = {Proceedings of the AAAI Symposium Series},
year = {2026},
volume = {10},
number = {1},
pages = {2--9},
doi = {10.1609/aaaiss.v10i1.43384}
}
EUVIMED