Large-scale Reinforcement Learning for Diffusion Models

2024-01-20 · Yinan Zhang, Eric Tzeng, Yilun Du, Dmitry Kislyuk

Text-to-image diffusion models are a class of deep generative models that have demonstrated an impressive capacity for high-quality image generation. However, these models are susceptible to implicit biases that arise from web-scale text-image training pairs and may inaccurately model aspects of images we care about. This can result in suboptimal samples, model bias, and images that do not align with human ethics and preferences. In this paper, we present an effective scalable algorithm to improve diffusion models using Reinforcement Learning (RL) across a diverse set of reward functions, such as human preference, compositionality, and fairness over millions of images. We illustrate how our approach substantially outperforms existing methods for aligning diffusion models with human preferences. We further illustrate how this substantially improves pretrained Stable Diffusion (SD) models, generating samples that are preferred by humans 80.3% of the time over those from the base SD model while simultaneously improving both the composition and diversity of generated samples.

📄 PDF Abstract BibTeX arXiv:2401.12244

Code (0)

등록된 구현이 없습니다.

Tasks

DiversityEthics Fairness Image Generation reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

BASE 설명 없음

Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets

2024-12-10 · Zhen Liu, Tim Z. Xiao, Weiyang Liu, Yoshua Bengio 외

While one commonly trains large diffusion models by collecting datasets on target downstream tasks, it is often desired to align and finetune pretrained diffusion models with some reward functions that are either designe…

Diversity

Multiscale Inverse Reinforcement Learning using Diffusion Wavelets

2016-11-24 · Jung-Su Ha, Han-Lim Choi

This work presents a multiscale framework to solve an inverse reinforcement learning (IRL) problem for continuous-time/state stochastic systems. We take advantage of a diffusion wavelet representation of the associated M…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces

2025-09-26 · Haitong Ma, Ofir Nabati, Aviv Rosenberg, Bo Dai 외 arxiv

Reinforcement learning (RL) struggles to scale to large, combinatorial action spaces common in many real-world problems. This paper introduces a novel framework for training discrete diffusion models as highly effective …

Reinforcement Learning

Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning

2025-02-03 · Hanyang Zhao, Haoxian Chen, Ji Zhang, David D. Yao 외

Reinforcement learning from human feedback (RLHF), which aligns a diffusion model with input prompt, has become a crucial step in building reliable generative AI models. Most works in this area use a discrete-time formul…

Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models

2025-09-08 · Yinjie Wang, Ling Yang, Bowen Li, Ye Tian 외 arxiv

We propose TraceRL, a trajectory-aware reinforcement learning framework for diffusion language models (DLMs) that incorporates preferred inference trajectory into post-training, and is applicable across different archite…

Reinforcement Learning Mathematical Reasoning