EUVIMEDEUVIMED entdecken
Crossref

SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences

Arpan Mukherjee, Marcello Bullo, Deniz Gündüz

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (44) · 37747-37755

DOI: 10.1609/aaai.v40i44.41110 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Arpan Mukherjee und 2 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Uniform-reward reinforcement learning from human feedback (RLHF), which trains a single reward model to represent the preferences of all annotators, fails to capture the diversity of opinions across sub-populations, inadvertently favoring dominant groups. The state-of-the-art, MaxMin-RLHF, addresses this by learning group-specific reward models, and by optimizing for the group receiving the minimum reward, thereby promoting fairness. However, we identify that a key limitation of MaxMin-RLHF is its poor performance when the minimum-reward group is a minority. To mitigate this drawback, we introduce a novel framework, termed *SharedRep-RLHF*. At its core, SharedRep-RLHF learns and leverages *shared preference traits* in annotations among various groups, in contrast to learning separate reward models across groups. We first show that MaxMin-RLHF is provably suboptimal in learning shared traits, and then quantify the sample complexity of SharedRep RLHF. Experiments across diverse natural language tasks showcase the effectiveness of ShareRep-RLHF compared to MaxMin-RLHF with a gain of up to 20% in win rate.

Zitieren

APA

Mukherjee, A., Bullo, M., & Gündüz, D. (2026). SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences. Proceedings of the AAAI Conference on Artificial Intelligence, 40(44), 37747-37755. https://doi.org/10.1609/aaai.v40i44.41110

Vancouver

Mukherjee A, Bullo M, Gündüz D. SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(44):37747-37755. doi:10.1609/aaai.v40i44.41110

BibTeX
@article{Mukherjee2026,
  author = {Mukherjee, Arpan and Bullo, Marcello and Gündüz, Deniz},
  title = {SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {44},
  pages = {37747--37755},
  doi = {10.1609/aaai.v40i44.41110}
}
Kostenlos starten →