EUVIMEDEUVIMED entdecken
Crossref

Learning Optimal Advantage from Preferences and Mistaking It for Reward

W. Bradley Knox, Stephane Hatgis-Kessell, Sigurdur Orn Adalgeirsson, Serena Booth, Anca Dragan, Peter Stone, Scott Niekum

Proceedings of the AAAI Conference on Artificial Intelligence · 24.03.2024 · 38 (9) · 10066-10073

DOI: 10.1609/aaai.v38i9.28870 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 24.03.2024. Verfasst von W. Bradley Knox und 6 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

We consider algorithms for learning reward functions from human preferences over pairs of trajectory segments, as used in reinforcement learning from human feedback (RLHF). Most recent work assumes that human preferences are generated based only upon the reward accrued within those segments, or their partial return. Recent work casts doubt on the validity of this assumption, proposing an alternative preference model based upon regret. We investigate the consequences of assuming preferences are based upon partial return when they actually arise from regret. We argue that the learned function is an approximation of the optimal advantage function, not a reward function. We find that if a specific pitfall is addressed, this incorrect assumption is not particularly harmful, resulting in a highly shaped reward function. Nonetheless, this incorrect usage of the approximation of the optimal advantage function is less desirable than the appropriate and simpler approach of greedy maximization of it. From the perspective of the regret preference model, we also provide a clearer interpretation of fine tuning contemporary large language models with RLHF. This paper overall provides insight regarding why learning under the partial return preference model tends to work so well in practice, despite it conforming poorly to how humans give preferences.

Zitieren

APA

Knox, W. B., Hatgis-Kessell, S., Adalgeirsson, S. O., Booth, S., Dragan, A., Stone, P., & Niekum, S. (2024). Learning Optimal Advantage from Preferences and Mistaking It for Reward. Proceedings of the AAAI Conference on Artificial Intelligence, 38(9), 10066-10073. https://doi.org/10.1609/aaai.v38i9.28870

Vancouver

Knox WB, Hatgis-Kessell S, Adalgeirsson SO, Booth S, Dragan A, Stone P, et al. Learning Optimal Advantage from Preferences and Mistaking It for Reward. Proceedings of the AAAI Conference on Artificial Intelligence. 2024;38(9):10066-10073. doi:10.1609/aaai.v38i9.28870

BibTeX
@article{Knox2024,
  author = {Knox, W. Bradley and Hatgis-Kessell, Stephane and Adalgeirsson, Sigurdur Orn and Booth, Serena and Dragan, Anca and Stone, Peter and Niekum, Scott},
  title = {Learning Optimal Advantage from Preferences and Mistaking It for Reward},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2024},
  volume = {38},
  number = {9},
  pages = {10066--10073},
  doi = {10.1609/aaai.v38i9.28870}
}
Kostenlos starten →