SGPO: Self-Generated Preference Optimization Based on Self-Improver
ACM Transactions on Intelligent Systems and Technology · 2026-08-31 · 17 (4) · 1-28
DOI: 10.1145/3800958 ↗At a glance
Journal article in ACM Transactions on Intelligent Systems and Technology, published online on 2026-05-18. Written by Hyeonji Lee and 3 co-authors. 34 references. The full text is freely available (open access, CC BY 4.0). Funded by: National Research Foundation of Korea, Institute of Information and Communications Technology Planning and Evaluation (IITP)-ITRC, Institute of Information and Communications Technology Planning and Evaluation.
Abstract (original language)
Large Language Models (LLMs) , despite their extensive pretraining on diverse datasets, require effective alignment to human preferences for practical and reliable deployment. Conventional alignment methods typically employ off-policy learning and depend on human-annotated datasets, which limits their broad applicability and introduces distribution shift issues during training. To address these challenges, we propose Self-Generated Preference Optimization based on Self-Improver (SGPO) , an innovative alignment framework that leverages an on-policy self-improving mechanism. Specifically, the improver refines responses from a policy model to self-generate preference data for Direct Preference Optimization (DPO) of the policy model. Here, the improver and policy are unified into a single model, and in order to generate higher-quality preference data, this self-improver learns to make incremental yet discernible improvements to the current responses by referencing supervised fine-tuning outputs. Experimental results on AlpacaEval 2.0 and Arena-Hard show that the proposed SGPO significantly improves performance over DPO and baseline self-improving methods without using external preference data.
Funding
- National Research Foundation of Korea
- Institute of Information and Communications Technology Planning and Evaluation (IITP)-ITRC
- Institute of Information and Communications Technology Planning and Evaluation
Cite
APA
Lee, H., Jo, D., Yun, S., & Kim, S. (2026). SGPO: Self-Generated Preference Optimization Based on Self-Improver. ACM Transactions on Intelligent Systems and Technology, 17(4), 1-28. https://doi.org/10.1145/3800958
Vancouver
Lee H, Jo D, Yun S, Kim S. SGPO: Self-Generated Preference Optimization Based on Self-Improver. ACM Transactions on Intelligent Systems and Technology. 2026;17(4):1-28. doi:10.1145/3800958
BibTeX
@article{Lee2026,
author = {Lee, Hyeonji and Jo, Daejin and Yun, Seohwan and Kim, Sungwoong},
title = {SGPO: Self-Generated Preference Optimization Based on Self-Improver},
journal = {ACM Transactions on Intelligent Systems and Technology},
year = {2026},
volume = {17},
number = {4},
pages = {1--28},
doi = {10.1145/3800958}
}
EUVIMED