EUVIMEDEUVIMED entdecken
Crossref

When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF

Yifan Xu, Xichen Ye, Yifan Chen, Qiaosheng Zhang

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (44) · 38057-38065

DOI: 10.1609/aaai.v40i44.41143 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Yifan Xu und 3 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Quality of datasets plays an important role in large language model (LLM) alignment. In collecting human feedback, however, preference flipping is ubiquitous and causes corruption in data annotation; the issue necessitates the alignment algorithms with improved robustness against potential flipped pairs. To this end, this paper introduces a Flipping-Aware Direct Preference Optimization (FA-DPO) algorithm tailored to preference flipping from a reinforcement learning with human feedback (RLHF) perspective. We dissect the inherent human intention model and the preference flipping mechanism introduced by external factors as two distinct stages; in the latter, we introduce an instance-dependent flipping probability on the basis of the Bradley-Terry (BT) model. Further, by leveraging features relevant to preference annotation, we capture uncertainty in judgments and model preference flipping patterns. In practice, we design a simple yet efficient iterative optimization algorithm compatible with the original RLHF and Direct Preference Optimization (DPO) algorithms. In our experiments, we investigate the instance-dependent preference flipping model under multiple circumstances for evaluation of our proposed method, as well as other baseline methods.

Zitieren

APA

Xu, Y., Ye, X., Chen, Y., & Zhang, Q. (2026). When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF. Proceedings of the AAAI Conference on Artificial Intelligence, 40(44), 38057-38065. https://doi.org/10.1609/aaai.v40i44.41143

Vancouver

Xu Y, Ye X, Chen Y, Zhang Q. When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(44):38057-38065. doi:10.1609/aaai.v40i44.41143

BibTeX
@article{Xu2026,
  author = {Xu, Yifan and Ye, Xichen and Chen, Yifan and Zhang, Qiaosheng},
  title = {When Human Preferences Flip: An Instance-Dependent Robust Loss for RLHF},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {44},
  pages = {38057--38065},
  doi = {10.1609/aaai.v40i44.41143}
}
Kostenlos starten →