KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration

2026-05-14 · Ruicheng Zhang, Kaixi Cong, Jun Zhou, Zhizhou Zhong, Zunnan Xu, Shuiyang Mao, Wei Liu, Xiu Li arxiv

Aligning streaming autoregressive (AR) video generators with human preferences is challenging. Existing reinforcement learning methods predominantly rely on noise-based exploration and SDE-based surrogate policies that are mismatched to the deterministic ODE dynamics of distilled AR models, and tend to perturb low-level appearance rather than the high-level semantic storyline progression critical for long-horizon coherence. To address these limitations, we present KVPO, an ODE-native online Group Relative Policy Optimization (GRPO) framework for aligning streaming video generators. For diversity exploration, KVPO introduces a causal-semantic exploration paradigm that relocates the source of variation from stochastic noise to the historical KV cache. By stochastically routing historical KV entries, it constructs semantically diverse generation branches that remain strictly on the data manifold. For policy modeling, KVPO introduces a velocity-field surrogate policy based on Trajectory Velocity Energy (TVE), which quantifies branch likelihood in flow-matching velocity space and yields a reward-weighted contrastive objective fully consistent with the native ODE formulation. Experiments on multiple distilled AR video generators demonstrate consistent gains in visual quality, motion quality, and text-video alignment across both single-prompt short-video and multi-prompt long-video settings.

📄 PDF Abstract BibTeX arXiv:2605.14278

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning Video Alignment

KVpop -- Key-Value Cache Compression with Predictive Online Pruning

2026-07-06 · Lukas Hauzenberger, Niklas Schmidinger, Anamaria-Roberta Hartl, David Stap 외 arxiv

Key-value (KV) cache growth is a major bottleneck in autoregressive decoding, as memory and bandwidth scale linearly with context length. Existing KV eviction methods often rely on static heuristics or proxy scores, whic…

Mathematical Reasoning

AR-CoPO: Align Autoregressive Video Generation with Contrastive Policy Optimization

2026-03-18 · Dailan He, Guanlin Feng, Xingtong Ge, Yi Zhang 외 arxiv

Streaming autoregressive (AR) video generators combined with few-step distillation achieve low-latency, high-quality synthesis, yet remain difficult to align via reinforcement learning from human feedback (RLHF). Existin…

Reinforcement Learning Domain Generalization Video Generation

Advances in GRPO for Generation Models: A Survey

2026-02-21 · Zexiang Liu, Xianglong He, Yangguang Li arxiv

Large-scale flow matching models have achieved strong performance across generative tasks such as text-to-image, video, 3D, and speech synthesis. However, aligning their outputs with human preferences and task-specific o…

Reinforcement Learning Speech Synthesis Video Generation Image Editing

InfLVG: Reinforce Inference-Time Consistent Long Video Generation with GRPO

2025-05-23 · Xueji Fang, Liyuan Ma, Zhiyang Chen, Mingyuan Zhou 외

Recent advances in text-to-video generation, particularly with autoregressive models, have enabled the synthesis of high-quality videos depicting individual scenes. However, extending these models to generate long, cross…

Text-to-Video Generation Video Generation

Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation

2025-10-15 · Yifu Luo, Xinhao Hu, Keyu Fan, Haoyuan Sun 외 arxiv

Reinforcement learning (RL) has garnered increasing attention in text-to-image (T2I) generation. However, most existing RL approaches are tailored to either diffusion models or autoregressive models, overlooking an impor…

Text-to-Image Generation Reinforcement Learning