Aligning Diffusion Models by Optimizing Human Utility
We present Diffusion-KTO, a novel approach for aligning text-to-image diffusion models by formulating the alignment objective as the maximization of expected human utility. Since this objective applies to each generation independently, Diffusion-KTO does not require collecting costly pairwise preference data nor training a complex reward model. Instead, our objective requires simple per-image binary feedback signals, e.g. likes or dislikes, which are abundantly available. After fine-tuning using Diffusion-KTO, text-to-image diffusion models exhibit superior performance compared to existing techniques, including supervised fine-tuning and Diffusion-DPO, both in terms of human judgment and automatic evaluation metrics such as PickScore and ImageReward. Overall, Diffusion-KTO unlocks the potential of leveraging readily available per-image binary signals and broadens the applicability of aligning text-to-image diffusion models with human preferences.
Code (2)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Step-level Denoising-time Diffusion Alignment with Multiple Objectives
Reinforcement learning (RL) has emerged as a powerful tool for aligning diffusion models with human preferences, typically by optimizing a single reward function under a KL regularization constraint. In practice, however…
Reinforcement LearningDifferentially Private Fine-Tuning of Diffusion Models
The integration of Differential Privacy (DP) with diffusion models (DMs) presents a promising yet challenging frontier, particularly due to the substantial memorization capabilities of DMs that pose significant privacy r…
Image GenerationMemorizationparameter-efficient fine-tuningSafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset
To mitigate the risk of harmful outputs from large vision models (LVMs), we introduce the SafeSora dataset to promote research on aligning text-to-video generation with human values. This dataset encompasses human prefer…
Safety AlignmentText-to-Video GenerationVideo AlignmentVideo GenerationD3S2: Diffusion-Guided Dataset Distillation for Semantic Segmentation
Dataset distillation (DD) aims to compress large-scale datasets into compact synthetic sets while preserving training efficacy. However, existing studies mainly focus on image classification, leaving dense prediction tas…
Semantic SegmentationImage ClassificationReinforcement Learning Beyond Expectation
The inputs and preferences of human users are important considerations in situations where these users interact with autonomous cyber or cyber-physical systems. In these scenarios, one is often interested in aligning beh…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)