EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

GapDPO: Setwise Direct Preference Optimization for LLM Safety Alignment

Hyeongsoo Lim, Jinyoung Kim, Min Ho Jang, Minji Cho, Eun Seo Seo, Ji Won Yoon

Applied Sciences · 03.11.2026 · 16 (21) · 10900

DOI: 10.3390/app162110900 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 03.11.2026. Verfasst von Hyeongsoo Lim und 5 weiteren Autor:innen. Die Arbeit nennt 38 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Institute of Information & Communications Technology Planning & Evaluation, Chung-Ang University, National Research Foundation of Korea.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

While reinforcement learning is widely used to align large language models (LLMs), its application to open-ended settings remains limited by reliance on preference datasets, making it difficult to scale to non-reasoning tasks such as safety. Applying direct preference optimization (DPO) to LLM safety alignment requires preferred and dispreferred responses for harmful instructions, making scalable preference construction a key bottleneck. We therefore propose GapDPO, a setwise preference optimization method that enables DPO-based safety alignment without reward models, verifiers, or preference annotation. GapDPO constructs preference sets by generating safe responses with our proposed refusal-style prompts and pairing them with affirmative targets from the dataset. GapDPO then maximizes the mean reward gap between the two sets while penalizing within-set variance. On AdvBench and HarmBench, GapDPO reduces the average attack success rate under PAIR, GCG, and AmpleGCG from 21.3% for refusal-style fine-tuning and 15.0% for single-pair DPO to 10.6%, while increasing strict set separation on training behaviors from 51% to as high as 63%. The improvement is mainly observed under GCG attacks and is accompanied by more refusals of benign requests. The averaged pairwise DPO on the same preference sets performs comparably, indicating that these sets account for most of the gain over single-pair DPO.

Förderung

  • Institute of Information & Communications Technology Planning & Evaluation
  • Chung-Ang University
  • National Research Foundation of Korea

Zitieren

APA

Lim, H., Kim, J., Jang, M. H., Cho, M., Seo, E. S., & Yoon, J. W. (2026). GapDPO: Setwise Direct Preference Optimization for LLM Safety Alignment. Applied Sciences, 16(21), 10900. https://doi.org/10.3390/app162110900

Vancouver

Lim H, Kim J, Jang MH, Cho M, Seo ES, Yoon JW. GapDPO: Setwise Direct Preference Optimization for LLM Safety Alignment. Applied Sciences. 2026;16(21):10900. doi:10.3390/app162110900

BibTeX
@article{Lim2026,
  author = {Lim, Hyeongsoo and Kim, Jinyoung and Jang, Min Ho and Cho, Minji and Seo, Eun Seo and Yoon, Ji Won},
  title = {GapDPO: Setwise Direct Preference Optimization for LLM Safety Alignment},
  journal = {Applied Sciences},
  year = {2026},
  volume = {16},
  number = {21},
  pages = {10900},
  doi = {10.3390/app162110900}
}
Kostenlos starten →