Towards General Preference Alignment: Diffusion Models at Nash Equilibrium
Reinforcement learning from human feedback (RLHF) has been popular for aligning text-to-image (T2I) diffusion models with human preferences. As a mainstream branch of RLHF, Direct Preference Optimization (DPO) offers a computationally efficient alternative that avoids explicit reward modeling and has been widely adopted in diffusion alignment. However, existing preference-based methods for diffusion alignment still rely on reward-induced preference signals and typically assume that human preferences can be adequately modeled by the Bradley--Terry (BT) model, which may fail to capture the full complexity of human preferences. In this paper, we formulate diffusion alignment from a game-theoretic perspective. We propose Diffusion Nash Preference Optimization (Diff.-NPO), an intuitive general preference framework for diffusion alignment. Diff.-NPO encourages the current policy to play against itself to achieve self improvement and lead to a better alignment. Empirically, we demonstrate the effectiveness of Diff.-NPO on the text-to-image generation task via various metrics. Diff.-NPO consistently outperforms existing preference-based diffusion alignment methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Text-to-Image GenerationReinforcement LearningSimilar Papers 제목 키워드 기반
COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferences
Many alignment methods, including reinforcement learning from human feedback (RLHF), rely on the Bradley-Terry reward assumption, which is insufficient to capture the full range of general human preferences. To achieve r…
Language ModelingLanguage ModellingHuman Alignment of Large Language Models through Online Preference Optimisation
Ensuring alignment of language models' outputs with human preferences is critical to guarantee a useful, safe, and pleasant user experience. Thus, human alignment has been extensively studied recently and several methods…
Multiplayer Nash Preference Optimization
Reinforcement learning from human feedback (RLHF) has emerged as the standard paradigm for aligning large language models with human preferences. However, reward-based methods grounded in the Bradley-Terry assumption str…
Reinforcement LearningFundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching
Nash Learning from Human Feedback is a game-theoretic framework for aligning large language models (LLMs) with human preferences by modeling learning as a two-player zero-sum game. However, using raw preference as the pa…
DiversityAsymptotic Equilibrium Analysis of the Boston Mechanism
We analyze the performance of the Boston mechanism under equilibrium play in uniform random markets. We provide two results. First, while the share of students assigned to their first preference is 63% under truthfulness…