What Makes Value Learning Efficient in Residual Reinforcement Learning?

2026-02-11 · Guozheng Ma, Lu Li, Haoyu Wang, Zixuan Liu, Pierre-Luc Bacon, Dacheng Tao arxiv

Residual reinforcement learning (RL) enables stable online refinement of expressive pretrained policies by freezing the base and learning only bounded corrections. However, value learning in residual RL poses unique challenges that remain poorly understood. In this work, we identify two key bottlenecks: cold start pathology, where the critic lacks knowledge of the value landscape around the base policy, and structural scale mismatch, where the residual contribution is dwarfed by the base action. Through systematic investigation, we uncover the mechanisms underlying these bottlenecks, revealing that simple yet principled solutions suffice: base-policy transitions serve as an essential value anchor for implicit warmup, and critic normalization effectively restores representation sensitivity for discerning value differences. Based on these insights, we propose DAWN (Data-Anchored Warmup and Normalization), a minimal approach targeting efficient value learning in residual RL. By addressing these bottlenecks, DAWN demonstrates substantial efficiency gains across diverse benchmarks, policy architectures, and observation modalities.

📄 PDF Abstract BibTeX arXiv:2602.10539

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

Deep Residual Reinforcement Learning

2019-05-03 · Shangtong Zhang, Wendelin Boehmer, Shimon Whiteson

We revisit residual algorithms in both model-free and model-based reinforcement learning settings. We propose the bidirectional target network technique to stabilize residual algorithms, yielding a residual version of DD…

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimism

2022-03-11 · Ming Yin, Yaqi Duan, Mengdi Wang, Yu-Xiang Wang

Offline reinforcement learning, which seeks to utilize offline/historical data to optimize sequential decision-making strategies, has gained surging prominence in recent studies. Due to the advantage that appropriate fun…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Bad-Policy Density: A Measure of Reinforcement Learning Hardness

2021-10-07 · David Abel, Cameron Allen, Dilip Arumugam, D. Ellis Hershkowitz 외

Reinforcement learning is hard in general. Yet, in many specific environments, learning is easy. What makes learning easy in one environment, but difficult in another? We address this question by proposing a simple measu…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Is the Bellman residual a bad proxy?

2016-06-24 · NeurIPS 2017 12 · Matthieu Geist, Bilal Piot, Olivier Pietquin

This paper aims at theoretically and empirically comparing two standard optimization criteria for Reinforcement Learning: i) maximization of the mean value and ii) minimization of the Bellman residual. For that purpose, …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

The Effect of Network Depth on the Optimization Landscape

2019-05-28 · Behrooz Ghorbani, Ying Xiao, Shankar Krishnan

It is well-known that deeper neural networks are harder to train than shallower ones. In this short paper, we use the (full) eigenvalue spectrum of the Hessian to explore how the loss landscape changes as the network get…