Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transport

2025-02-18 · Mingyang Sun, Pengxiang Ding, Weinan Zhang, Donglin Wang

Diffusion policies have shown promise in learning complex behaviors from demonstrations, particularly for tasks requiring precise control and long-term planning. However, they face challenges in robustness when encountering distribution shifts. This paper explores improving diffusion-based imitation learning models through online interactions with the environment. We propose OTPR (Optimal Transport-guided score-based diffusion Policy for Reinforcement learning fine-tuning), a novel method that integrates diffusion policies with RL using optimal transport theory. OTPR leverages the Q-function as a transport cost and views the policy as an optimal transport map, enabling efficient and stable fine-tuning. Moreover, we introduce masked optimal transport to guide state-action matching using expert keypoints and a compatibility-based resampling strategy to enhance training stability. Experiments on three simulation tasks demonstrate OTPR's superior performance and robustness compared to existing methods, especially in complex and sparse-reward environments. In sum, OTPR provides an effective framework for combining IL and RL, achieving versatile and reliable policy learning. The code will be released at https://github.com/Sunmmyy/OTPR.git.

📄 PDF Abstract BibTeX arXiv:2502.12631

Code (1)

sunmmyy/otpr 공식 구현 pytorch

Tasks

Imitation Learning

Methods 이 논문이 사용한 방법론

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Efficient Diffusion Policies for Offline Reinforcement Learning

2023-05-31 · NeurIPS 2023 11 · Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang 외

Offline reinforcement learning (RL) aims to learn optimal policies from offline datasets, where the parameterization of policies is crucial but often overlooked. Recently, Diffsuion-QL significantly boosts the performanc…

D4RL Offline RL Policy Gradient Methods reinforcement-learning+2

Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning

2024-07-29 · Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang 외

One important property of DIstribution Correction Estimation (DICE) methods is that the solution is the optimal stationary distribution ratio between the optimized and data collection policy. In this work, we show that D…

Offline RL reinforcement-learning Reinforcement Learning

RL Dreams: Policy Gradient Optimization for Score Distillation based 3D Generation

2023-12-08 · Aradhya N. Mathur, Phu Pham, Aniket Bera, Ojaswa Sharma

3D generation has rapidly accelerated in the past decade owing to the progress in the field of generative modeling. Score Distillation Sampling (SDS) based rendering has improved 3D asset generation to a great extent. Fu…

3D Generation Denoising Policy Gradient Methods Text to 3D

Policy Transfer for Continuous-Time Reinforcement Learning: A (Rough) Differential Equation Approach

2025-10-16 · Xin Guo, Zijiu Lyu arxiv

This paper studies policy transfer, one of the well-known transfer learning techniques adopted in large language models, for continuous-time reinforcement learning problems. In the case of continuous-time linear-quadrati…

Reinforcement Learning Transfer Learning

Flow Matching Policy Gradients

2025-07-28 · David McAllister, Songwei Ge, Brent Yi, Chung Min Kim 외 arxiv

Flow-based generative models, including diffusion models, excel at modeling continuous distributions in high-dimensional spaces. In this work, we introduce Flow Policy Optimization (FPO), a simple on-policy reinforcement…

Reinforcement Learning Continuous Control