Pixel-wise RL on Diffusion Models: Reinforcement Learning from Rich Feedback

2024-04-05 · Mo Kordzanganeh, Danial Keshvary, Nariman Arian

Latent diffusion models are the state-of-the-art for synthetic image generation. To align these models with human preferences, training the models using reinforcement learning on human feedback is crucial. Black et. al 2024 introduced denoising diffusion policy optimisation (DDPO), which accounts for the iterative denoising nature of the generation by modelling it as a Markov chain with a final reward. As the reward is a single value that determines the model's performance on the entire image, the model has to navigate a very sparse reward landscape and so requires a large sample count. In this work, we extend the DDPO by presenting the Pixel-wise Policy Optimisation (PXPO) algorithm, which can take feedback for each pixel, providing a more nuanced reward to the model.

📄 PDF Abstract BibTeX arXiv:2404.04356

Code (0)

등록된 구현이 없습니다.

Tasks

Denoising Image GenerationNavigatereinforcement-learning

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

ALIGN In the ALIGN method, visual and language representations are jointly trained from noisy image alt-text data. The image and text encoders are learned via contrastive loss…

Towards Better Optimization For Listwise Preference in Diffusion Models

2025-10-02 · Jiamu Bai, Xin Yu, Meilong Xu, Weitao Lu 외 arxiv

Reinforcement learning from human feedback (RLHF) has proven effectiveness for aligning text-to-image (T2I) diffusion models with human preferences. Although Direct Preference Optimization (DPO) is widely adopted for its…

Text-to-Image Generation Computational Efficiency Reinforcement Learning Image Editing

Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models

2026-05-26 · Austin Wang, Jiaqi Han, Stefano Ermon, Yisong Yue arxiv

Preference optimization has emerged as an efficient alternative to online reinforcement learning from human feedback (RLHF) for aligning text-to-image diffusion models. However, existing methods largely reduce supervisio…

Text-to-Image Generation Reinforcement Learning Image Editing

Rich Human Feedback for Text-to-Image Generation

2023-12-15 · CVPR 2024 1 · Youwei Liang, Junfeng He, Gang Li, Peizhao Li 외

Recent Text-to-Image (T2I) generation models such as Stable Diffusion and Imagen have made significant progress in generating high-resolution images based on text descriptions. However, many generated images still suffer…

Image Generation Text to Image Generation Text-to-Image Generation

StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback

2025-10-23 · Jiho Park, Sieun Choi, Jaeyoon Seo, Jihie Kim arxiv

Although recent advancements in diffusion models have significantly enriched the quality of generated images, challenges remain in synthesizing pixel-based human-drawn sketches, a representative example of abstract expre…

Visual Question Answering Reinforcement Learning

Arena as Offline Reward: Efficient Fine-Grained Preference Optimization for Diffusion Models

2026-05-07 · Zhikai Li, Yue Zhao, Edward Zhongwei Zhang, Xuewen Liu 외 arxiv

Reinforcement learning from human feedback (RLHF) effectively promotes preference alignment of text-to-image (T2I) diffusion models. To improve computational efficiency, direct preference optimization (DPO), which avoids…

Computational Efficiency Reinforcement Learning