홈 › Papers

FlowDPG: Deterministic Policy Gradient on Flow Matching Policies for Real-World Manipulation

2026-06-21 · Kexin Shi, Junyao Shi, Poorvi Hebbar, Zhuolun Zhao, Tarun Amarnath, Yifan Su, Shikhar Bahl, Deepak Pathak arxiv

Real-world reinforcement learning for robotic manipulation remains challenging, and this difficulty is amplified for flow matching policies: applying policy gradient methods to these policies is fundamentally limited by the need to backpropagate through time(BPTT) along the multi-step ODE that maps noise to actions, which is computationally prohibitive and numerically fragile. We propose FlowDPG, a DDPG-style method specifically designed for flow matching policies that distills the critic gradient into the velocity field at training time, bypassing BPTT entirely. Intuitively, FlowDPG combines two complementary vectors: the demonstration-driven velocity that keeps the action feasible, and the critic-driven correction that steers it toward higher value. Our contributions are threefold: (1) a BPTT-free distillation framework that enables stable DDPG-style policy improvement on flow matching policies, (2) a formal connection between the FlowDPG update direction and vanilla Deterministic Policy Gradient via three explicit approximations, and (3) real-world validation on a long-horizon, multi-stage, dual-arm AirPods assembly task, where FlowDPG attains a 92% end-to-end success rate, substantially outperforming recent RL methods spanning value-conditioning, auxiliary-module adaptation, and adjoint-based critic-gradient approaches. Videos and more results are provided on the project page https://flowdpg.github.io.

📄 PDF Abstract BibTeX arXiv:2606.22303

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

2026-08-27 · Shiyi Zhang, Mushui Liu, Yunze Tong, Wanggui He 외 hf

On-policy distillation (OPD), which leverages a pre-trained, specialized teacher model to provide dense supervisory signals, has achieved significant success in Large Language Models (LLMs) and has recently been adapted …

Improved techniques for fine-tuning flow models via adjoint matching: a deterministic control pipeline

2026-05-07 · Zhengyi Guo, Jiayuan Sheng, David D. Yao, Wenpin Tang arxiv

We propose a deterministic adjoint matching framework that formulates human preference alignment for flow-based generative models as an optimal control problem over velocity fields. One can directly regress the control t…

ReFPO: Reflow Regularization for Flow Matching Policy Gradients

2026-06-19 · Ge Wang, Yibo Peng, Fan Feng, Shenhao Yan 외 arxiv

We present Reflow-regularized Flow Matching Policy Gradients (ReFPO), a simple online RL method that adds explicit Reflow regularization to FPO for efficient flow-based control. We uncover a key structural property: the …

Physical Simulations

GROW: Group-Relative Advantage-Weighted On-Policy Reinforcement Learning of Autoregressive-Diffusion Text-to-Speech model

2026-08-04 · Guanrou Yang, Tian Tan, Qian Chen, Ziyang Ma 외 arxiv

Reinforcement learning for flow-matching text-to-speech is complicated by deterministic ODE sampling: trajectory-level policy-gradient methods typically convert the ODE into an SDE and track per-step likelihood ratios, i…

Reinforcement Learning

Flow Matching Policy Gradients

2025-07-28 · David McAllister, Songwei Ge, Brent Yi, Chung Min Kim 외 arxiv

Flow-based generative models, including diffusion models, excel at modeling continuous distributions in high-dimensional spaces. In this work, we introduce Flow Policy Optimization (FPO), a simple on-policy reinforcement…

Reinforcement Learning Continuous Control