Grad2Reward: From Sparse Judgment to Dense Rewards for Improving Open-Ended LLM Reasoning

2026-02-02 · Zheng Zhang, Ao Lu, Yuanhao Zeng, Ziwei Shan, Jinjin Guo, Lufei Li, Yexin Li, Kan Ren arxiv

Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant breakthroughs in complex LLM reasoning within verifiable domains, such as mathematics and programming. Recent efforts have sought to extend this paradigm to open-ended tasks by employing LLMs-as-a-Judge to provide sequence-level rewards for policy optimization. However, these rewards are inherently sparse, failing to provide the fine-grained supervision necessary for generating complex, long-form trajectories. Furthermore, current work treats the Judge as a black-box oracle, discarding the rich intermediate feedback signals encoded in it. To address these limitations, we introduce Grad2Reward, a novel framework that extracts dense process rewards directly from the Judge's model inference process via a single backward pass. By leveraging gradient-based attribution, Grad2Reward enables precise token-level credit assignment, substantially enhancing training efficiency and reasoning quality. Additionally, Grad2Reward introduces a self-judging mechanism, allowing the policy to improve through its own evaluative signals without training specialized reward models or reliance on superior external Judges. The experiments demonstrate that policies optimized with Grad2Reward achieve outstanding performance across diverse open-ended tasks, affirming its effectiveness and broad generalizability.

📄 PDF Abstract BibTeX arXiv:2602.01791

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

A Study on Dense and Sparse (Visual) Rewards in Robot Policy Learning

2021-08-06 · Abdalkarim Mohtasib, Gerhard Neumann, Heriberto Cuayahuitl

Deep Reinforcement Learning (DRL) is a promising approach for teaching robots new behaviour. However, one of its main limitations is the need for carefully hand-coded reward signals by an expert. We argue that it is cruc…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

$f$-Policy Gradients: A General Framework for Goal Conditioned RL using $f$-Divergences

2023-10-10 · Siddhant Agarwal, Ishan Durugkar, Peter Stone, Amy Zhang

Goal-Conditioned Reinforcement Learning (RL) problems often have access to sparse rewards where the agent receives a reward signal only when it has achieved the goal, making policy optimization a difficult problem. Sever…

Efficient Exploration Policy Gradient Methods Reinforcement Learning (RL)

MemBuilder: Reinforcing LLMs for Long-Term Memory Construction via Attributed Dense Rewards

2026-01-09 · Zhiyu Shen, Ziming Wu, Fuming Lai, Shaobing Lian 외 arxiv

Maintaining consistency in long-term dialogues remains a fundamental challenge for LLMs, as standard retrieval mechanisms often fail to capture the temporal evolution of historical states. While memory-augmented framewor…

Reinforcement Learning Question Generation

f-Policy Gradients: A General Framework for Goal-Conditioned RL using f-Divergences

2023-09-21 · NeurIPS 2023 11

Learning Guidance Rewards with Trajectory-space Smoothing

2020-10-23 · NeurIPS 2020 12 · Tanmay Gangwani, Yuan Zhou, Jian Peng

Long-term temporal credit assignment is an important challenge in deep reinforcement learning (RL). It refers to the ability of the agent to attribute actions to consequences that may occur after a long time interval. Ex…

Attribute Deep Reinforcement Learning Q-Learning Reinforcement Learning (RL)