홈 › Papers

Policy Representation via Diffusion Probability Model for Reinforcement Learning

2023-05-22 · Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, Zhouchen Lin

Popular reinforcement learning (RL) algorithms tend to produce a unimodal policy distribution, which weakens the expressiveness of complicated policy and decays the ability of exploration. The diffusion probability model is powerful to learn complicated multimodal distributions, which has shown promising and potential applications to RL. In this paper, we formally build a theoretical foundation of policy representation via the diffusion probability model and provide practical implementations of diffusion policy for online model-free RL. Concretely, we character diffusion policy as a stochastic process, which is a new approach to representing a policy. Then we present a convergence guarantee for diffusion policy, which provides a theory to understand the multimodality of diffusion policy. Furthermore, we propose the DIPO which is an implementation for model-free online RL with DIffusion POlicy. To the best of our knowledge, DIPO is the first algorithm to solve model-free online RL problems with the diffusion model. Finally, extensive empirical results show the effectiveness and superiority of DIPO on the standard continuous control Mujoco benchmark.

📄 PDF Abstract BibTeX arXiv:2305.13122

Code (1)

bellmantimehut/dipo 공식 구현 pytorch

Tasks

continuous-control Continuous ControlMuJoCoreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning

2023-10-09 · Longxiang He, Li Shen, Linrui Zhang, Junbo Tan 외

Constrained policy search (CPS) is a fundamental problem in offline reinforcement learning, which is generally solved by advantage weighted regression (AWR). However, previous methods may still encounter out-of-distribut…

D4RL Offline RL reinforcement-learning

Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning

2024-02-06 · Ruoqi Zhang, Ziwei Luo, Jens Sjölund, Thomas B. Schön 외

This paper presents advanced techniques of training diffusion policies for offline reinforcement learning (RL). At the core is a mean-reverting stochastic differential equation (SDE) that transfers a complex action distr…

D4RL Offline RL reinforcement-learning Reinforcement Learning (RL)

Diffusion Policy through Conditional Proximal Policy Optimization

2026-03-05 · Ben Liu, Shunpeng Yang, Hua Chen arxiv

Reinforcement learning (RL) has been extensively employed in a wide range of decision-making problems, such as games and robotics. Recently, diffusion policies have shown strong potential in modeling multi-modal behavior…

Reinforcement Learning

A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models

2026-07-16 · Zikun Zhang, Jiayuan Sheng, David D. Yao, Wenpin Tang arxiv

We formulate reinforcement learning (RL) in continuous time with discrete state spaces and possibly arbitrary action spaces via a stochastic control approach, where the state dynamics are modeled as a controlled continuo…

Mathematical Reasoning Reinforcement Learning

SP-GCRL: Influence Maximization on Incomplete Social Graphs

2026-03-31 · Haohua Niu, Yuxuan Yang, Lingfeng Zhang, Hao Li 외 arxiv

Influence maximization (IM) in real platforms is challenged by incomplete, noisy social graphs and non-stationary diffusion dynamics. We propose SP-GCRL, a social-propagation-aware graph contrastive reinforcement learnin…

Reinforcement Learning Contrastive Learning