EUVIMEDEUVIMED entdecken
Crossref

When Alignment Erases Disagreement: Preference Collapse and a Four-Valued Alternative for LLMs

Aizierjiang Aiersilan

Proceedings of the AAAI Symposium Series · 06.11.2026 · 10 (1) · 2-9

DOI: 10.1609/aaaiss.v10i1.43384 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Symposium Series, online erschienen am 06.11.2026. Verfasst von Aizierjiang Aiersilan.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Alignment pipelines for large language models convert human judgments into a scalar reward and optimize a single policy against it. We argue that this conversion, not merely the demography of the annotator pool, makes contemporary alignment structurally insensitive to Global South, Indigenous, and other marginalized value frameworks. We present two results. First, under the Bradley--Terry model that underwrites RLHF and DPO, preference labels collected without annotator provenance from a deeply divided population and from a uniformly ambivalent one have identical distributions, so no estimator can distinguish a contested question from one about which users are indifferent. Second, the evaluation space that separates support, opposition, ignorance, and conflict is a three-dimensional simplex whose vertices are the four values of Belnap--Dunn logic; a scalar reward is the output of a map with two-dimensional fibers, which discards conflict entirely and folds ignorance evenly into support and opposition. Consensus-seeking elicitation does not repair the collapse; it removes contested items by construction. We therefore propose a four-valued architecture that returns admissible sets rather than maximizers, maintains a non-negotiable safety floor in the classical fragment while treating contested values paraconsistently, and exposes contestation to institutional governance instead of resolving it silently; we analyze its attack surface, use Sen's liberal paradox as a diagnostic map of the design space, and propose conflict-preserving evaluation metrics.

Zitieren

APA

Aiersilan, A. (2026). When Alignment Erases Disagreement: Preference Collapse and a Four-Valued Alternative for LLMs. Proceedings of the AAAI Symposium Series, 10(1), 2-9. https://doi.org/10.1609/aaaiss.v10i1.43384

Vancouver

Aiersilan A. When Alignment Erases Disagreement: Preference Collapse and a Four-Valued Alternative for LLMs. Proceedings of the AAAI Symposium Series. 2026;10(1):2-9. doi:10.1609/aaaiss.v10i1.43384

BibTeX
@article{Aiersilan2026,
  author = {Aiersilan, Aizierjiang},
  title = {When Alignment Erases Disagreement: Preference Collapse and a Four-Valued Alternative for LLMs},
  journal = {Proceedings of the AAAI Symposium Series},
  year = {2026},
  volume = {10},
  number = {1},
  pages = {2--9},
  doi = {10.1609/aaaiss.v10i1.43384}
}
Kostenlos starten →