SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (44) · 37747-37755
DOI: 10.1609/aaai.v40i44.41110 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Arpan Mukherjee und 2 weiteren Autor:innen.
Abstract (Originalsprache)
Uniform-reward reinforcement learning from human feedback (RLHF), which trains a single reward model to represent the preferences of all annotators, fails to capture the diversity of opinions across sub-populations, inadvertently favoring dominant groups. The state-of-the-art, MaxMin-RLHF, addresses this by learning group-specific reward models, and by optimizing for the group receiving the minimum reward, thereby promoting fairness. However, we identify that a key limitation of MaxMin-RLHF is its poor performance when the minimum-reward group is a minority. To mitigate this drawback, we introduce a novel framework, termed *SharedRep-RLHF*. At its core, SharedRep-RLHF learns and leverages *shared preference traits* in annotations among various groups, in contrast to learning separate reward models across groups. We first show that MaxMin-RLHF is provably suboptimal in learning shared traits, and then quantify the sample complexity of SharedRep RLHF. Experiments across diverse natural language tasks showcase the effectiveness of ShareRep-RLHF compared to MaxMin-RLHF with a gain of up to 20% in win rate.
Zitieren
APA
Mukherjee, A., Bullo, M., & Gündüz, D. (2026). SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences. Proceedings of the AAAI Conference on Artificial Intelligence, 40(44), 37747-37755. https://doi.org/10.1609/aaai.v40i44.41110
Vancouver
Mukherjee A, Bullo M, Gündüz D. SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(44):37747-37755. doi:10.1609/aaai.v40i44.41110
BibTeX
@article{Mukherjee2026,
author = {Mukherjee, Arpan and Bullo, Marcello and Gündüz, Deniz},
title = {SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {44},
pages = {37747--37755},
doi = {10.1609/aaai.v40i44.41110}
}
EUVIMED