EUVIMEDOpdag EUVIMED
Open access · CC BY 4.0

SGPO: Self-Generated Preference Optimization Based on Self-Improver

Hyeonji Lee, Daejin Jo, Seohwan Yun, Sungwoong Kim

ACM Transactions on Intelligent Systems and Technology · 2026-08-31 · 17 (4) · 1-28

DOI: 10.1145/3800958 ↗

At a glance

Journal article in ACM Transactions on Intelligent Systems and Technology, published online on 2026-05-18. Written by Hyeonji Lee and 3 co-authors. 34 references. The full text is freely available (open access, CC BY 4.0). Funded by: National Research Foundation of Korea, Institute of Information and Communications Technology Planning and Evaluation (IITP)-ITRC, Institute of Information and Communications Technology Planning and Evaluation.

AbstractBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Large Language Models (LLMs) , despite their extensive pretraining on diverse datasets, require effective alignment to human preferences for practical and reliable deployment. Conventional alignment methods typically employ off-policy learning and depend on human-annotated datasets, which limits their broad applicability and introduces distribution shift issues during training. To address these challenges, we propose Self-Generated Preference Optimization based on Self-Improver (SGPO) , an innovative alignment framework that leverages an on-policy self-improving mechanism. Specifically, the improver refines responses from a policy model to self-generate preference data for Direct Preference Optimization (DPO) of the policy model. Here, the improver and policy are unified into a single model, and in order to generate higher-quality preference data, this self-improver learns to make incremental yet discernible improvements to the current responses by referencing supervised fine-tuning outputs. Experimental results on AlpacaEval 2.0 and Arena-Hard show that the proposed SGPO significantly improves performance over DPO and baseline self-improving methods without using external preference data.

Funding

  • National Research Foundation of Korea
  • Institute of Information and Communications Technology Planning and Evaluation (IITP)-ITRC
  • Institute of Information and Communications Technology Planning and Evaluation

Cite

APA

Lee, H., Jo, D., Yun, S., & Kim, S. (2026). SGPO: Self-Generated Preference Optimization Based on Self-Improver. ACM Transactions on Intelligent Systems and Technology, 17(4), 1-28. https://doi.org/10.1145/3800958

Vancouver

Lee H, Jo D, Yun S, Kim S. SGPO: Self-Generated Preference Optimization Based on Self-Improver. ACM Transactions on Intelligent Systems and Technology. 2026;17(4):1-28. doi:10.1145/3800958

BibTeX
@article{Lee2026,
  author = {Lee, Hyeonji and Jo, Daejin and Yun, Seohwan and Kim, Sungwoong},
  title = {SGPO: Self-Generated Preference Optimization Based on Self-Improver},
  journal = {ACM Transactions on Intelligent Systems and Technology},
  year = {2026},
  volume = {17},
  number = {4},
  pages = {1--28},
  doi = {10.1145/3800958}
}
Start for free →