Skip-Connected Policy Optimization for Implicit Advantage
Group Relative Policy Optimization (GRPO) has proven effective in RLVR by using outcome-based rewards. While fine-grained dense rewards can theoretically improve performance, we reveal that under practical sampling budgets, Monte Carlo estimation yields high-variance and sign-inconsistent advantages for early reasoning tokens, paradoxically underperforming outcome-only GRPO. We propose Skip-Connected Optimization (SKPO), which decomposes reasoning into upstream and downstream phases: upstream receives dense rewards from downstream Monte Carlo sampling with single-stream optimization; downstream maintains group-relative optimization, where a skip connection concatenates the upstream segment with the original problem, enabling the model to leverage helpful upstream reasoning while preserving the freedom to bypass flawed reasoning through direct problem access. Experiments demonstrate improvements of 3.91% and 6.17% relative gains over the strongest baselines on Qwen2.5-Math-7B and Llama-3.2-3B respectively across mathematical benchmarks and out-of-domain tasks including general reasoning and code generation. Further analysis reveals an implicit advantage: SKPO generates trajectories with higher intermediate-step quality even when matched for final correctness.
Code (0)
등록된 구현이 없습니다.
Tasks
Code GenerationSimilar Papers 제목 키워드 기반
Implicit Euler Skip Connections: Enhancing Adversarial Robustness via Numerical Stability
Deep neural networks have achieved great success in various areas. However, recent works have found that neural networks are vulnerable to adversarial attacks, which leads to a hot topic nowadays. Although many approache…
Adversarial AttackAdversarial RobustnessAlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimization
Implicit Q-learning (IQL) serves as a strong baseline for offline RL, which learns the value function using only dataset actions through quantile regression. However, it is unclear how to recover the implicit policy from…
D4RLOffline RLQ-Learningquantile regression+1Skip-connected 3D DenseNet for volumetric infant brain MRI segmentation
Automatic 6-month infant brain tissue segmentation of magnetic resonance imaging (MRI) is still less accurate owing to the low intensity contrast among tissues. To tackle the problem, we introduce an accurate segmentatio…
Infant Brain Mri SegmentationMRI segmentationSegmentationVIMPO: Value-Implicit Policy Optimization for LLMs
Reinforcement learning with verifiable rewards has become a central tool for improving the reasoning ability of large language models, but current methods face a trade-off between simplicity and credit assignment. Group-…
Reinforcement LearningOptimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth
Graph Neural Networks (GNNs) have been studied through the lens of expressive power and generalization. However, their optimization properties are less well understood. We take the first step towards analyzing GNN traini…