Training Diffusion Models Towards Diverse Image Generation with Reinforcement Learning

2024-01-01 · CVPR 2024 1 · Zichen Miao, Jiang Wang, Ze Wang, Zhengyuan Yang, Lijuan Wang, Qiang Qiu, Zicheng Liu

Diffusion models have demonstrated unprecedented capabilities in image generation. Yet they incorporate and amplify the data bias (e.g. gender age) from the original training set limiting the diversity of generated images. In this paper we propose a diversity-oriented fine-tuning method using reinforcement learning (RL) for diffusion models under the guidance of an image-set-based reward function. Specifically the proposed reward function denoted as Diversity Reward utilizes a set of generated images to evaluate the coverage of the current generative distribution w.r.t. the reference distribution represented by a set of unbiased images. Built on top of the probabilistic method of distribution discrepancy estimation Diversity Reward can measure the relative distribution gap with a small set of images efficiently. We further formulate the diffusion process as a multi-step decision-making problem (MDP) and apply policy gradient methods to fine-tune diffusion models by maximizing the Diversity Reward. The proposed rewards are validated on a post-sampling selection task where a subset of the most diverse images are selected based on Diversity Reward values. We also show the effectiveness of our RL fine-tuning framework on enhancing the diversity of image generation with different types of diffusion models including class-conditional models and text-conditional models e.g. StableDiffusion.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Decision MakingDiversityImage GenerationPolicy Gradient Methodsreinforcement-learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Large-scale Reinforcement Learning for Diffusion Models

2024-01-20 · Yinan Zhang, Eric Tzeng, Yilun Du, Dmitry Kislyuk

Text-to-image diffusion models are a class of deep generative models that have demonstrated an impressive capacity for high-quality image generation. However, these models are susceptible to implicit biases that arise fr…

Diversity Ethics Fairness Image Generation+3

Enhancing Diffusion Models with Text-Encoder Reinforcement Learning

2023-11-27 · Chaofeng Chen, Annan Wang, HaoNing Wu, Liang Liao 외

Text-to-image diffusion models are typically trained to optimize the log-likelihood objective, which presents challenges in meeting specific requirements for downstream tasks, such as image aesthetics and image-text alig…

reinforcement-learning Reinforcement Learning

LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Models

2026-02-15 · Shufan Li, Yuchen Zhu, Jiuxiang Gu, Kangning Liu 외 arxiv

Diffusion language models (dLLMs) recently emerged as a promising alternative to auto-regressive LLMs. The latest works further extended it to multimodal understanding and generation tasks. In this work, we propose LaViD…

Reinforcement Learning Image Editing

DanceGRPO: Unleashing GRPO on Visual Generation

2025-05-12 · Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong 외

Recent breakthroughs in generative models-particularly diffusion models and rectified flows-have revolutionized visual content creation, yet aligning model outputs with human preferences remains a critical challenge. Exi…

Denoising reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

HP-Edit: A Human-Preference Post-Training Framework for Image Editing

2026-04-21 · Fan Li, Chonghuinan Wang, Lina Lei, Yuping Qiu 외 arxiv

Common image editing tasks typically adopt powerful generative diffusion models as the leading paradigm for real-world content editing. Meanwhile, although reinforcement learning (RL) methods such as Diffusion-DPO and Fl…