VAMPO: Policy Optimization for Improving Visual Dynamics in Video Action Models
Video action models are an appealing foundation for Vision--Language--Action systems because they can learn visual dynamics from large-scale video data and transfer this knowledge to downstream robot control. Yet current diffusion-based video predictors are trained with likelihood-surrogate objectives, which encourage globally plausible predictions without explicitly optimizing the precision-critical visual dynamics needed for manipulation. This objective mismatch often leads to subtle errors in object pose, spatial relations, and contact timing that can be amplified by downstream policies. We propose VAMPO, a post-training framework that directly improves visual dynamics in video action models through policy optimization. Our key idea is to formulate multi-step denoising as a sequential decision process and optimize the denoising policy with rewards defined over expert visual dynamics in latent space. To make this optimization practical, we introduce an Euler Hybrid sampler that injects stochasticity only at the first denoising step, enabling tractable low-variance policy-gradient estimation while preserving the coherence of the remaining denoising trajectory. We further combine this design with GRPO and a verifiable non-adversarial reward. Across diverse simulated and real-world manipulation tasks, VAMPO improves task-relevant visual dynamics, leading to better downstream action generation and stronger generalization. The homepage is https://vampo-robot.github.io/VAMPO/.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Pre-trained Visual Dynamics Representations for Efficient Policy Learning
Pre-training for Reinforcement Learning (RL) with purely video data is a valuable yet challenging problem. Although in-the-wild videos are readily available and inhere a vast amount of prior world knowledge, the absence …
Reinforcement Learning (RL)Video PredictionWorld KnowledgeVideo-Enhanced Offline Reinforcement Learning: A Model-Based Approach
Offline reinforcement learning (RL) enables policy optimization using static datasets, avoiding the risks and costs of extensive real-world exploration. However, it struggles with suboptimal offline behaviors and inaccur…
Autonomous DrivingOffline RLreinforcement-learningReinforcement Learning+1GigaWorld-Policy: An Efficient Action-Centered World--Action Model
World-Action Models (WAM) initialized from pre-trained video generation backbones have demonstrated remarkable potential for robot policy learning. However, existing approaches face two critical bottlenecks that hinder p…
Video GenerationImagined Value Gradients: Model-Based Policy Optimization with Transferable Latent Dynamics Models
Humans are masters at quickly learning many complex tasks, relying on an approximate understanding of the dynamics of their environments. In much the same way, we would like our learning agents to quickly adapt to new ta…
Model-based Reinforcement LearningReinforcement LearningReinforcement Learning (RL)Robot Manipulation+1Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
Visual representations play a crucial role in developing generalist robotic policies. Previous vision encoders, typically pre-trained with single-image reconstruction or two-image contrastive learning, tend to capture st…
Contrastive LearningImage ReconstructionRobot ManipulationVideo Generation+1