Adaptive KL Control for Direct Preference Optimization in Instruction-Following LLMs
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (48) · 41480-41482
DOI: 10.1609/aaai.v40i48.42313 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Yi Khuen Chai.
Abstract (Originalsprache)
The scaling parameter β in Direct Preference Optimization governs a fundamental trade-off: low β produces weak gradients that fail to learn from ambiguous preferences, while high β amplifies updates and causes excessive drift from the reference policy. Prior work treats β as fixed or scheduled throughout training. We introduce DualLoop-DPO, which modulates β via dual feedback: a fast loop raises β temporarily on high-uncertainty batches to enforce stronger preference margins, while a slow loop uses EMA-smoothed KL tracking to regulate policy drift. Experiments on preference alignment benchmarks show consistent improvements over existing static-β, β-scheduling, and dynamic-β baselines. These findings suggest that dual-loop β control—responding to uncertainty for learning and divergence for stability—offers a promising direction for preference-based fine-tuning.
Zitieren
APA
Chai, Y. K. (2026). Adaptive KL Control for Direct Preference Optimization in Instruction-Following LLMs. Proceedings of the AAAI Conference on Artificial Intelligence, 40(48), 41480-41482. https://doi.org/10.1609/aaai.v40i48.42313
Vancouver
Chai YK. Adaptive KL Control for Direct Preference Optimization in Instruction-Following LLMs. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(48):41480-41482. doi:10.1609/aaai.v40i48.42313
BibTeX
@article{Chai2026,
author = {Chai, Yi Khuen},
title = {Adaptive KL Control for Direct Preference Optimization in Instruction-Following LLMs},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {48},
pages = {41480--41482},
doi = {10.1609/aaai.v40i48.42313}
}
EUVIMED