Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models

2026-06-09 · Bowen Ping, Xiangxin Zhou, Penghui Qi, Minnan Luo, Liefeng Bo, Tianyu Pang arxiv

Recent work has demonstrated that online reinforcement learning (RL) can substantially improve the quality and alignment of flow matching models for image and video generation. Methods such as Flow-GRPO and CPS cast the denoising process as a Markov Decision Process and apply PPO-style ratio clipping to enforce a trust region. However, we argue that ratio clipping is structurally ill-suited for flow models: the probability ratio between new and old policies is a noisy, single-sample estimate of the true policy divergence, leading to over-constraining in some regions of the trajectory and under-constraining in others. We propose Flow-DPPO (Flow Divergence Proximal Policy Optimization), which replaces ratio clipping with a divergence proximal constraint. A key observation is that the per-step policy in flow models is Gaussian, enabling exact and cheap computation of the KL divergence between old and new policies. Flow-DPPO employs an asymmetric divergence mask that blocks gradient updates only when they simultaneously move away from the trusted region and violate the divergence threshold. Experiments show that Flow-DPPO achieves higher rewards with better KL-proximal efficiency, alleviates catastrophic forgetting, promotes balanced multi-objective optimization, and enables stable multi-epoch training where ratio clipping degrades. Code and models are available at https://github.com/Tencent-Hunyuan/UniRL/tree/main/FlowDPPO.

📄 PDF Abstract BibTeX arXiv:2606.11025

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Video Generation

Rethinking the Trust Region in LLM Reinforcement Learning

2026-02-04 · Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang 외 arxiv

Reinforcement learning (RL) has become a cornerstone for fine-tuning Large Language Models (LLMs), with Proximal Policy Optimization (PPO) serving as the de facto standard algorithm. Despite its ubiquity, we argue that t…

Reinforcement Learning

Train What You Deploy:Token-Faithful Post-Training of a Production Coding

2026-09-04 · Cheng Li, Jiexiong Liu, Yixuan Chen, Chi Hong arxiv

Existing post-training pipelines for coding and terminal agents suffer severe token and control fidelity errors: simplified training environments mismatch production deployments, and offline token reconstruction from age…

A Reinforcement Learning Method for Environments with Stochastic Variables: Post-Decision Proximal Policy Optimization with Dual Critic Networks

2025-04-07 · Leonardo Kanashiro Felizardo, Edoardo Fadda, Paolo Brandimarte, Emilio Del-Moral-Hernandez 외

This paper presents Post-Decision Proximal Policy Optimization (PDPPO), a novel variation of the leading deep reinforcement learning method, Proximal Policy Optimization (PPO). The PDPPO state transition process is divid…

Deep Reinforcement Learning

Rethinking the Divergence Regularization in LLM RL

2026-06-08 · Jiarui Yao, Xiangxin Zhou, Penghui Qi, Wee Sun Lee 외 arxiv

Reinforcement learning (RL) has become a key component of post-training large language models (LLMs). In practice, LLM RL is often off-policy because of training-inference mismatch and policy staleness, making trust-regi…

Reinforcement Learning

Proximal Policy Optimization with Mixed Distributed Training

2019-07-15 · Zhen-Yu Zhang, Xiangfeng Luo, Tong Liu, Shaorong Xie 외

Instability and slowness are two main problems in deep reinforcement learning. Even if proximal policy optimization (PPO) is the state of the art, it still suffers from these two problems. We introduce an improved algori…

Deep Reinforcement Learning Reinforcement Learning