Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
Reinforcement learning (RL) has emerged as a powerful paradigm for enhancing reasoning capabilities in large language models, but faces a fundamental asymmetry in computation and memory requirements: inference is embarrassingly parallel with a minimal memory footprint, while policy updates require extensive synchronization and are memory-intensive. To address this asymmetry, we introduce PODS (Policy Optimization with Down-Sampling), a framework that strategically decouples these phases by generating numerous rollouts in parallel but updating only on an informative subset. Within this framework, we develop max-variance down-sampling, a theoretically motivated method that selects rollouts with maximally diverse reward signals. We prove that this approach has an efficient algorithmic solution, and empirically demonstrate that GRPO with PODS using max-variance down-sampling achieves superior performance over standard GRPO on the GSM8K benchmark.
Code (0)
등록된 구현이 없습니다.
Tasks
AllGSM8KReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Parameter Exploration for RLVR via Variational Learning
Exploration has been a focus of reinforcement learning research for a long time. Recently, there has been growing evidence that it is also an important ingredient in LLM reinforcement learning recipes that can significan…
Mathematical ReasoningReinforcement LearningCode GenerationMaximum Entropy Model Rollouts: Fast Model Based Policy Optimization without Compounding Errors
Model usage is the central challenge of model-based reinforcement learning. Although dynamics model based on deep neural networks provide good generalization for single step prediction, such ability is over exploited whe…
modelModel-based Reinforcement Learningreinforcement-learningReinforcement Learning+1FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning
Reinforcement learning has become a cornerstone for aligning and unlocking the reasoning capabilities of large-scale models. At its core, the training loop of GRPO and its variants alternates between rollout sampling and…
Reinforcement LearningFAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning
Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for enhancing the reasoning capabilities of large language models (LLMs). In this context, models explore reasoning trajectories a…
Reinforcement LearningGroup Adaptive Clipping Policy Optimization
Group relative policy optimization for reinforcement learning with verifiable rewards (RLVR) typically uses a fixed importance-sampling (IS) ratio clipping boundary across all rollouts. We identify a key limitation: rare…
Reinforcement Learning