EUVIMEDEUVIMED entdecken
Übersichtsarbeit

A Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications

Wenyi Xiao, Zechuan Wang, Leilei Gan, Shuai Zhao, Zongyue Li, Ruirui Lei, Wanggui He, Luu Anh Tuan, Long Chan, Hao Jiang, Zhou Zhao, Fei Wu

IEEE Transactions on Pattern Analysis and Machine Intelligence · 2026

DOI: 10.1109/tpami.2026.3704314 ↗

Kurz gefasst

Übersichtsarbeit in der Zeitschrift IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026. Verfasst von Wenyi Xiao und 11 weiteren Autor:innen. Hauptthemen laut MEDLINE: Large Language Models. Die Arbeit nennt 193 Literaturangaben. Gefördert von: Ningbo Youth Science and Technology Innovation Leading Talent Program, Pioneer and Leading Goose” R&D Program of Zhejiang, Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

With the rapid advancement of large language models (LLMs), aligning policy models with human preferences has become increasingly critical. Direct Preference Optimization (DPO) has emerged as a promising approach for alignment, acting as an RL-free alternative to Reinforcement Learning from Human Feedback (RLHF). Despite DPO's various advancements and inherent limitations, an in-depth review of these aspects is currently lacking in the literature. In this work, we present a review of the challenges and opportunities in DPO, covering theoretical analyses, variants, relevant preference datasets, and applications. Specifically, we categorize recent studies on DPO based on key research questions to provide a thorough understanding of DPO's current landscape. Additionally, we propose several future research directions to offer insights on model alignment for the research community.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsHumansAlgorithmsReinforcement Machine Learning

Förderung

  • Ningbo Youth Science and Technology Innovation Leading Talent Program
  • Pioneer and Leading Goose” R&D Program of Zhejiang
  • Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies
  • Science and Technology Project of State Grid Beijing Electric Power Company
  • Risk Identification Technology Based on Soft Robots and Artificial Intelligence

Zitieren

APA

Xiao, W., Wang, Z., Gan, L., Zhao, S., Li, Z., Lei, R., He, W., Tuan, L. A., Chan, L., Jiang, H., Zhao, Z., & Wu, F. (2026). A Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2026.3704314

Vancouver

Xiao W, Wang Z, Gan L, Zhao S, Li Z, Lei R, et al. A Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2026. doi:10.1109/tpami.2026.3704314

BibTeX
@article{Xiao2026,
  author = {Xiao, Wenyi and Wang, Zechuan and Gan, Leilei and Zhao, Shuai and Li, Zongyue and Lei, Ruirui and He, Wanggui and Tuan, Luu Anh and Chan, Long and Jiang, Hao and Zhao, Zhou and Wu, Fei},
  title = {A Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2026},
  doi = {10.1109/tpami.2026.3704314}
}
Kostenlos starten →