TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inference

2025-09-18 · Dan Zhang, Min Cai, Jonathan Light, Ziniu Hu, Yisong Yue, Jie Tang arxiv

Reward models are central to both reinforcement learning (RL) with language models and inference-time verification. However, existing reward models often lack temporal consistency, leading to ineffective policy updates and unstable RL training. We introduce TDRM, a method for learning smoother and more reliable reward models by minimizing temporal differences (TD) for training-time reinforcement learning and inference-time verification. Experiments show that TD-trained process reward models (PRMs) improve performance across Best-of-N (up to 6.6%) and tree-search (up to 23.7%) settings. When combined with Reinforcement Learning with Verifiable Rewards (RLVR), TD-trained PRMs lead to more data-efficient RL -- achieving comparable performance with just 2.5k data to what baseline methods require 50.1k data to attain -- and yield higher-quality language model policies in 8 model variants (5 series), e.g., Qwen2.5-(0.5B, 1,5B), GLM4-9B-0414, GLM-Z1-9B-0414, Qwen2.5-Math-(1.5B, 7B), and DeepSeek-R1-Distill-Qwen-(1.5B, 7B). We release all code at https://github.com/THUDM/TDRM.

📄 PDF Abstract BibTeX arXiv:2509.15110

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Online Inference in Distributional Temporal-Difference Learning

2026-08-14 · Yang Peng, Liangyu Zhang arxiv

We study online statistical inference for functionals of the return distribution under a fixed policy. The return distribution is estimated by nonparametric distributional temporal-difference learning from a single Marko…

Learning Dynamics and Generalization in Reinforcement Learning

2022-06-05 · Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska 외

Solving a reinforcement learning (RL) problem poses two competing challenges: fitting a potentially discontinuous value function, and generalizing well to new observations. In this paper, we analyze the learning dynamics…

Policy Gradient Methods reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

In Hindsight: A Smooth Reward for Steady Exploration

2019-06-24 · Hadi S. Jomaa, Josif Grabocka, Lars Schmidt-Thieme

In classical Q-learning, the objective is to maximize the sum of discounted rewards through iteratively using the Bellman equation as an update, in an attempt to estimate the action value function of the optimal policy. …

Atari Games Q-Learning

Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models

2025-12-30 · Lars van der Laan, Aurelien Bibaut, Nathan Kallus arxiv

In many sequential decision-making problems, researchers observe actions but not the rewards that drive behavior, yet still wish to evaluate and compare counterfactual policies. Inverse reinforcement learning (IRL) and d…

Reinforcement Learning

DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothing

2023-11-02 · Vint Lee, Pieter Abbeel, Youngwoon Lee

Model-based reinforcement learning (MBRL) has gained much attention for its ability to learn complex behaviors in a sample-efficient way: planning actions by generating imaginary trajectories with predicted rewards. Desp…

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning