EUVIMEDEUVIMED entdecken
Crossref

Adaptive KL Control for Direct Preference Optimization in Instruction-Following LLMs

Yi Khuen Chai

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (48) · 41480-41482

DOI: 10.1609/aaai.v40i48.42313 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Yi Khuen Chai.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The scaling parameter β in Direct Preference Optimization governs a fundamental trade-off: low β produces weak gradients that fail to learn from ambiguous preferences, while high β amplifies updates and causes excessive drift from the reference policy. Prior work treats β as fixed or scheduled throughout training. We introduce DualLoop-DPO, which modulates β via dual feedback: a fast loop raises β temporarily on high-uncertainty batches to enforce stronger preference margins, while a slow loop uses EMA-smoothed KL tracking to regulate policy drift. Experiments on preference alignment benchmarks show consistent improvements over existing static-β, β-scheduling, and dynamic-β baselines. These findings suggest that dual-loop β control—responding to uncertainty for learning and divergence for stability—offers a promising direction for preference-based fine-tuning.

Zitieren

APA

Chai, Y. K. (2026). Adaptive KL Control for Direct Preference Optimization in Instruction-Following LLMs. Proceedings of the AAAI Conference on Artificial Intelligence, 40(48), 41480-41482. https://doi.org/10.1609/aaai.v40i48.42313

Vancouver

Chai YK. Adaptive KL Control for Direct Preference Optimization in Instruction-Following LLMs. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(48):41480-41482. doi:10.1609/aaai.v40i48.42313

BibTeX
@article{Chai2026,
  author = {Chai, Yi Khuen},
  title = {Adaptive KL Control for Direct Preference Optimization in Instruction-Following LLMs},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {48},
  pages = {41480--41482},
  doi = {10.1609/aaai.v40i48.42313}
}
Kostenlos starten →