Blockwise Advantage Estimation for Multi-Objective RL with Verifiable Rewards

2026-02-10 · Kirill Pavlenko, Alexander Golubev, Simon Karasik, Boris Yangel arxiv

Group Relative Policy Optimization (GRPO) assigns a single scalar advantage to all tokens in a completion. For structured generations with explicit segments and objectives, this couples unrelated reward signals across segments, leading to objective interference and misattributed credit. We propose Blockwise Advantage Estimation, a family of GRPO-compatible methods that assigns each objective its own advantage and applies it only to the tokens in the corresponding text block, reducing reliance on hand-designed scalar rewards and scaling naturally to additional objectives. A key challenge is estimating advantages for later blocks whose rewards are conditioned on sampled prefixes; standard unbiased approaches require expensive nested rollouts from intermediate states. Concretely, we introduce an Outcome-Conditioned Baseline that approximates intermediate state values using only within-group statistics by stratifying samples according to a prefix-derived intermediate outcome. On math tasks with uncertainty estimation, our method mitigates reward interference, is competitive with a state-of-the-art reward-designed approach, and preserves test-time gains from confidence-weighted ensembling. More broadly, it provides a modular recipe for optimizing sequential objectives in structured generations without additional rollouts.

📄 PDF Abstract BibTeX arXiv:2602.10231

Code (0)

등록된 구현이 없습니다.

MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization

2026-01-12 · Yang Zhao, Hepeng Wang, Xiao Ding, Yangou Ouyang 외 arxiv

Group-Relative Policy Optimization (GRPO) has emerged as an efficient paradigm for aligning Large Language Models (LLMs), yet its efficacy is primarily confined to domains with verifiable ground truths. Extending GRPO to…

Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning

2021-12-10 · Giseung Park, Sungho Choi, Youngchul Sung

This paper proposes a new sequential model learning architecture to solve partially observable Markov decision problems. Rather than compressing sequential information at every timestep as in conventional recurrent neura…

Partially Observable Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards

2026-06-27 · Yupeng Chang, Yuan Wu, Yi Chang arxiv

Critic-free reinforcement learning with verifiable rewards (RLVR), exemplified by Group Relative Policy Optimization (GRPO), avoids training a value function (critic) and reduces memory and compute overhead relative to c…

Reinforcement Learning

On the Trainability of Masked Diffusion Language Models via Blockwise Locality

2026-04-27 · Yuxiang Wang, Yu Xiang, Baojian Zhou, Qifang Zhao 외 arxiv

Masked diffusion language models (MDMs) have recently emerged as a promising alternative to standard autoregressive large language models (AR-LLMs), yet their optimization can be substantially less stable. We study block…

CoRPO: Adding a Correctness Bias to GRPO Improves Generalization

2025-11-06 · Anisha Garg, Claire Zhang, Nishit Neema, David Bick 외 arxiv

Group-Relative Policy Optimization (GRPO) has emerged as the standard for training reasoning capabilities in large language models through reinforcement learning. By estimating advantages using group-mean rewards rather …