홈 › Papers

Diffusion Policy Policy Optimization

2024-09-01 · Allen Z. Ren, Justin Lidard, Lars L. Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, Max Simchowitz

We introduce Diffusion Policy Policy Optimization, DPPO, an algorithmic framework including best practices for fine-tuning diffusion-based policies (e.g. Diffusion Policy) in continuous control and robot learning tasks using the policy gradient (PG) method from reinforcement learning (RL). PG methods are ubiquitous in training RL policies with other policy parameterizations; nevertheless, they had been conjectured to be less efficient for diffusion-based policies. Surprisingly, we show that DPPO achieves the strongest overall performance and efficiency for fine-tuning in common benchmarks compared to other RL methods for diffusion-based policies and also compared to PG fine-tuning of other policy parameterizations. Through experimental investigation, we find that DPPO takes advantage of unique synergies between RL fine-tuning and the diffusion parameterization, leading to structured and on-manifold exploration, stable training, and strong policy robustness. We further demonstrate the strengths of DPPO in a range of realistic settings, including simulated robotic tasks with pixel observations, and via zero-shot deployment of simulation-trained policies on robot hardware in a long-horizon, multi-stage manipulation task. Website with code: diffusion-ppo.github.io

📄 PDF Abstract BibTeX arXiv:2409.00588

Code (1)

irom-princeton/dppo pytorch

Tasks

continuous-control Continuous Control Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Forward KL Regularized Preference Optimization for Aligning Diffusion Policies

2024-09-09 · Zhao Shan, Chenyou Fan, Shuang Qiu, Jiyuan Shi 외

Diffusion models have achieved remarkable success in sequential decision-making by leveraging the highly expressive model capabilities in policy learning. A central problem for learning diffusion policies is to align the…

D4RL Decision Making Reinforcement Learning (RL)Sequential Decision Making

DiPOD: Diffusion Policy Optimization without Drifting Apart

2026-06-11 · Haozhe Jiang, Haiwen Feng, Pieter Abbeel, Jiantao Jiao 외 arxiv

RL post-training has become increasingly pivotal for improving diffusion policies, but existing diffusion policy-gradient methods are often unstable and cannot achieve reliable policy improvement. We identify the cause a…

Diffusion-Augmented Markov Decision Processes for Maximum Entropy Reinforcement Learning

2025-12-01 · Sebastian Sanokowski, Kaustubh Patil arxiv

Diffusion models excel at sampling from complex, unnormalized distributions. In this work, we extend Maximum Entropy Reinforcement Learning (ME-RL) to diffusion processes, enabling sampling from the optimal policy trajec…

Reinforcement Learning

Scaling World-Model Reinforcement Learning Through Diffusion Policy Optimization

2026-05-25 · Xiaoyuan Cheng, Wenxuan Yuan, Zhancun Mu, Yuanzhao Zhang 외 arxiv

Model-based reinforcement learning (RL) can be effectively supported at scale through the use of world models. However, in practice, scaling such approaches remains fundamentally limited. A commonly recognized challenge …