EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

SGPO: Self-Generated Preference Optimization Based on Self-Improver

Hyeonji Lee, Daejin Jo, Seohwan Yun, Sungwoong Kim

ACM Transactions on Intelligent Systems and Technology · 31.08.2026 · 17 (4) · 1-28

DOI: 10.1145/3800958 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 18.05.2026. Verfasst von Hyeonji Lee und 3 weiteren Autor:innen. Die Arbeit nennt 34 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Research Foundation of Korea, Institute of Information and Communications Technology Planning and Evaluation (IITP)-ITRC, Institute of Information and Communications Technology Planning and Evaluation.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large Language Models (LLMs) , despite their extensive pretraining on diverse datasets, require effective alignment to human preferences for practical and reliable deployment. Conventional alignment methods typically employ off-policy learning and depend on human-annotated datasets, which limits their broad applicability and introduces distribution shift issues during training. To address these challenges, we propose Self-Generated Preference Optimization based on Self-Improver (SGPO) , an innovative alignment framework that leverages an on-policy self-improving mechanism. Specifically, the improver refines responses from a policy model to self-generate preference data for Direct Preference Optimization (DPO) of the policy model. Here, the improver and policy are unified into a single model, and in order to generate higher-quality preference data, this self-improver learns to make incremental yet discernible improvements to the current responses by referencing supervised fine-tuning outputs. Experimental results on AlpacaEval 2.0 and Arena-Hard show that the proposed SGPO significantly improves performance over DPO and baseline self-improving methods without using external preference data.

Förderung

  • National Research Foundation of Korea
  • Institute of Information and Communications Technology Planning and Evaluation (IITP)-ITRC
  • Institute of Information and Communications Technology Planning and Evaluation

Zitieren

APA

Lee, H., Jo, D., Yun, S., & Kim, S. (2026). SGPO: Self-Generated Preference Optimization Based on Self-Improver. ACM Transactions on Intelligent Systems and Technology, 17(4), 1-28. https://doi.org/10.1145/3800958

Vancouver

Lee H, Jo D, Yun S, Kim S. SGPO: Self-Generated Preference Optimization Based on Self-Improver. ACM Transactions on Intelligent Systems and Technology. 2026;17(4):1-28. doi:10.1145/3800958

BibTeX
@article{Lee2026,
  author = {Lee, Hyeonji and Jo, Daejin and Yun, Seohwan and Kim, Sungwoong},
  title = {SGPO: Self-Generated Preference Optimization Based on Self-Improver},
  journal = {ACM Transactions on Intelligent Systems and Technology},
  year = {2026},
  volume = {17},
  number = {4},
  pages = {1--28},
  doi = {10.1145/3800958}
}
Kostenlos starten →