EUVIMEDEUVIMED entdecken
Crossref

Bootstrapping LLMs via Preference-Based Policy Optimization

Chen Jia

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (37) · 31256-31264

DOI: 10.1609/aaai.v40i37.40388 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Chen Jia.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Bootstrapping large language models (LLMs) via preference-based policy optimization enables aligning model behavior with human preferences while reducing reliance on extensive manual annotations. We propose a novel preference-based policy optimization (PbPO) framework that formulates learning as a min-max game between the LLM policy and a reward model (RM). The RM is constrained within a confidence set derived from collected preferences to ensure reliable exploitation, while simultaneously promoting robust exploration. Our iterative online algorithm actively collects new preference data from the evolving policy, enabling continual self-improvement of both the policy and the RM. We provide theoretical guarantees, establishing high-probability regret bounds for both sequence-level and token-level RMs. Extensive experiments across five benchmark datasets demonstrate that PbPO consistently outperforms state-of-the-art preference optimization methods.

Zitieren

APA

Jia, C. (2026). Bootstrapping LLMs via Preference-Based Policy Optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 40(37), 31256-31264. https://doi.org/10.1609/aaai.v40i37.40388

Vancouver

Jia C. Bootstrapping LLMs via Preference-Based Policy Optimization. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(37):31256-31264. doi:10.1609/aaai.v40i37.40388

BibTeX
@article{Jia2026,
  author = {Jia, Chen},
  title = {Bootstrapping LLMs via Preference-Based Policy Optimization},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {37},
  pages = {31256--31264},
  doi = {10.1609/aaai.v40i37.40388}
}
Kostenlos starten →