EUVIMEDEUVIMED entdecken
Crossref

Dual Active Learning for Reinforcement Learning from Human Feedback

Pangpang Liu, Chengchun Shi, Will Wei Sun

Operations Research · 20.11.2026

DOI: 10.1287/opre.2025.1837 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Operations Research, online erschienen am 20.11.2026. Verfasst von Pangpang Liu und 2 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Aligning large language models (LLMs) with human preferences is critical to recent advances in generative artificial intelligence. Reinforcement learning from human feedback (RLHF) is widely applied to achieve this objective. A key step in RLHF is to learn the reward function from human feedback. However, human feedback is costly and time-consuming, making it essential to collect high-quality conversation data for human teachers to label. Additionally, different human teachers have different levels of expertise. It is thus critical to query the most appropriate teacher for their opinions. In this paper, we use offline reinforcement learning (RL) to formulate the alignment problem. Motivated by the idea of $D$-optimal design, we first propose a dual active reward learning algorithm for the simultaneous selection of conversations and teachers. Next, we apply pessimistic RL to solve the alignment problem, based on the learned reward estimator. Theoretically, we show that the reward estimator obtained through our proposed adaptive selection strategy achieves minimal generalized variance asymptotically, and prove that the sub-optimality of our pessimistic policy scales as $O(1/\sqrt{T})$ with a given sample budget $T$. Through simulations and experiments on LLMs, we demonstrate the effectiveness of our algorithm and its superiority over state-of-the-art.

Zitieren

APA

Liu, P., Shi, C., & Sun, W. W. (2026). Dual Active Learning for Reinforcement Learning from Human Feedback. Operations Research. https://doi.org/10.1287/opre.2025.1837

Vancouver

Liu P, Shi C, Sun WW. Dual Active Learning for Reinforcement Learning from Human Feedback. Operations Research. 2026. doi:10.1287/opre.2025.1837

BibTeX
@article{Liu2026,
  author = {Liu, Pangpang and Shi, Chengchun and Sun, Will Wei},
  title = {Dual Active Learning for Reinforcement Learning from Human Feedback},
  journal = {Operations Research},
  year = {2026},
  doi = {10.1287/opre.2025.1837}
}
Kostenlos starten →