Bootstrapping LLMs via Preference-Based Policy Optimization
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (37) · 31256-31264
DOI: 10.1609/aaai.v40i37.40388 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Chen Jia.
Abstract (Originalsprache)
Bootstrapping large language models (LLMs) via preference-based policy optimization enables aligning model behavior with human preferences while reducing reliance on extensive manual annotations. We propose a novel preference-based policy optimization (PbPO) framework that formulates learning as a min-max game between the LLM policy and a reward model (RM). The RM is constrained within a confidence set derived from collected preferences to ensure reliable exploitation, while simultaneously promoting robust exploration. Our iterative online algorithm actively collects new preference data from the evolving policy, enabling continual self-improvement of both the policy and the RM. We provide theoretical guarantees, establishing high-probability regret bounds for both sequence-level and token-level RMs. Extensive experiments across five benchmark datasets demonstrate that PbPO consistently outperforms state-of-the-art preference optimization methods.
Zitieren
APA
Jia, C. (2026). Bootstrapping LLMs via Preference-Based Policy Optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 40(37), 31256-31264. https://doi.org/10.1609/aaai.v40i37.40388
Vancouver
Jia C. Bootstrapping LLMs via Preference-Based Policy Optimization. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(37):31256-31264. doi:10.1609/aaai.v40i37.40388
BibTeX
@article{Jia2026,
author = {Jia, Chen},
title = {Bootstrapping LLMs via Preference-Based Policy Optimization},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {37},
pages = {31256--31264},
doi = {10.1609/aaai.v40i37.40388}
}
EUVIMED