Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning

2025-02-07 · Chen-Xiao Gao, Chenyang Wu, Mingjun Cao, Chenjun Xiao, Yang Yu, Zongzhang Zhang

The primary focus of offline reinforcement learning (RL) is to manage the risk of hazardous exploitation of out-of-distribution actions. An effective approach to achieve this goal is through behavior regularization, which augments conventional RL objectives by incorporating constraints that enforce the policy to remain close to the behavior policy. Nevertheless, existing literature on behavior-regularized RL primarily focuses on explicit policy parameterizations, such as Gaussian policies. Consequently, it remains unclear how to extend this framework to more advanced policy parameterizations, such as diffusion models. In this paper, we introduce BDPO, a principled behavior-regularized RL framework tailored for diffusion-based policies, thereby combining the expressive power of diffusion policies and the robustness provided by regularization. The key ingredient of our method is to calculate the Kullback-Leibler (KL) regularization analytically as the accumulated discrepancies in reverse-time transition kernels along the diffusion trajectory. By integrating the regularization, we develop an efficient two-time-scale actor-critic RL algorithm that produces the optimal policy while respecting the behavior constraint. Comprehensive evaluations conducted on synthetic 2D tasks and continuous control tasks from the D4RL benchmark validate its effectiveness and superior performance.

📄 PDF Abstract BibTeX arXiv:2502.04778

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous Control D4RL reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Focus 설명 없음

Score Regularized Policy Optimization through Diffusion Behavior

2023-10-11 · Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su 외

Recent developments in offline reinforcement learning have uncovered the immense potential of diffusion modeling, which excels at representing heterogeneous behavior policies. However, sampling from diffusion policies is…

D4RL

Model-Based Offline Meta-Reinforcement Learning with Regularization

2022-02-07 · ICLR 2022 4 · Sen Lin, Jialin Wan, Tengyu Xu, Yingbin Liang 외

Existing offline reinforcement learning (RL) methods face a few major challenges, particularly the distributional shift between the learned policy and the behavior policy. Offline Meta-RL is emerging as a promising appro…

Meta Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Policy-Guided Diffusion

2024-04-09 · Matthew Thomas Jackson, Michael Tryfan Matthews, Cong Lu, Benjamin Ellis 외

In many real-world settings, agents must learn from an offline dataset gathered by some prior behavior policy. Such a setting naturally leads to distribution shift between the behavior policy and the target policy being …

Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning

2024-02-06 · Ruoqi Zhang, Ziwei Luo, Jens Sjölund, Thomas B. Schön 외

This paper presents advanced techniques of training diffusion policies for offline reinforcement learning (RL). At the core is a mean-reverting stochastic differential equation (SDE) that transfers a complex action distr…

D4RL Offline RL reinforcement-learning Reinforcement Learning (RL)

Forward KL Regularized Preference Optimization for Aligning Diffusion Policies

2024-09-09 · Zhao Shan, Chenyou Fan, Shuang Qiu, Jiyuan Shi 외

Diffusion models have achieved remarkable success in sequential decision-making by leveraging the highly expressive model capabilities in policy learning. A central problem for learning diffusion policies is to align the…

D4RL Decision Making Reinforcement Learning (RL)Sequential Decision Making