What Can RL Bring to VLA Generalization? An Empirical Study
Large Vision-Language Action (VLA) models have shown significant potential for embodied AI. However, their predominant training via supervised fine-tuning (SFT) limits generalization due to susceptibility to compounding errors under distribution shifts. Reinforcement learning (RL) offers a path to overcome these limitations by optimizing for task objectives via trial-and-error, yet a systematic understanding of its specific generalization benefits for VLAs compared to SFT is lacking. To address this, our study introduces a comprehensive benchmark for evaluating VLA generalization and systematically investigates the impact of RL fine-tuning across diverse visual, semantic, and execution dimensions. Our extensive experiments reveal that RL fine-tuning, particularly with PPO, significantly enhances generalization in semantic understanding and execution robustness over SFT, while maintaining comparable visual robustness. We identify PPO as a more effective RL algorithm for VLAs than LLM-derived methods like DPO and GRPO. We also develop a simple recipe for efficient PPO training on VLAs, and demonstrate its practical utility for improving VLA generalization. The project page is at https://rlvla.github.io
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement Learning (RL)Vision-Language-ActionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
What About Inputing Policy in Value Function: Policy Representation and Policy-extended Value Function Approximator
We study Policy-extended Value Function Approximator (PeVFA) in Reinforcement Learning (RL), which extends conventional value function approximator (VFA) to take as input not only the state (and action) but also an expli…
continuous-controlContinuous ControlContrastive LearningOpenAI Gym+2Understanding The Role of Adversarial Regularization in Supervised Learning
Despite numerous attempts sought to provide empirical evidence of adversarial regularization outperforming sole supervision in various inverse problems, the theoretical understanding of such phenomena remains elusive. In…
Shedding a PAC-Bayesian Light on Adaptive Sliced-Wasserstein Distances
The Sliced-Wasserstein distance (SW) is a computationally efficient and theoretically grounded alternative to the Wasserstein distance. Yet, the literature on its statistical properties -- or, more accurately, its genera…
Generalization BoundsUnderstanding the Role of Adversarial Regularization in Supervised Learning
Despite numerous attempts sought to provide empirical evidence of adversarial regularization outperforming sole supervision, the theoretical understanding of such phenomena remains elusive. In this study, we aim to resol…
The Price of Implicit Bias in Adversarially Robust Generalization
We study the implicit bias of optimization in robust empirical risk minimization (robust ERM) and its connection with robust generalization. In classification settings under adversarial perturbations with linear models, …