Representations for Stable Off-Policy Reinforcement Learning

2020-07-10 · ICML 2020 1 · Dibya Ghosh, Marc G. Bellemare

Reinforcement learning with function approximation can be unstable and even divergent, especially when combined with off-policy learning and Bellman updates. In deep reinforcement learning, these issues have been dealt with empirically by adapting and regularizing the representation, in particular with auxiliary tasks. This suggests that representation learning may provide a means to guarantee stability. In this paper, we formally show that there are indeed nontrivial state representations under which the canonical TD algorithm is stable, even when learning off-policy. We analyze representation learning schemes that are based on the transition matrix of a policy, such as proto-value functions, along three axes: approximation error, stability, and ease of estimation. In the most general case, we show that a Schur basis provides convergence guarantees, but is difficult to estimate from samples. For a fixed reward function, we find that an orthogonal basis of the corresponding Krylov subspace is an even better choice. We conclude by empirically demonstrating that these stable representations can be learned using stochastic gradient descent, opening the door to improved techniques for representation learning with deep networks.

📄 PDF Abstract BibTeX arXiv:2007.05520

Code (0)

등록된 구현이 없습니다.

Tasks

Deep Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Representation Learning

Stable Offline Value Function Learning with Bisimulation-based Representations

2024-10-02 · Brahma S. Pavse, Yudong Chen, Qiaomin Xie, Josiah P. Hanna

In reinforcement learning, offline value function learning is the procedure of using an offline dataset to estimate the expected discounted return from each state when taking actions according to a fixed target policy. T…

Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States

2026-05-08 · Yunho Choi, Jongwon Lim, Woojin Ahn, Minjae Oh 외 arxiv

Reinforcement learning with verifiable rewards (RLVR) for Large Reasoning Models hinges on baseline estimation for variance reduction, but existing approaches pay a heavy price: PPO requires a policy-model scale critic, …

Reinforcement Learning

When Object-Centric World Models Meet Policy Learning: From Pixels to Policies, and Where It Breaks

2025-11-08 · Stefano Ferraro, Akihiro Nakano, Masahiro Suzuki, Yutaka Matsuo arxiv

Object-centric world models (OCWM) aim to decompose visual scenes into object-level representations, providing structured abstractions that could improve compositional generalization and data efficiency in reinforcement …

Reinforcement Learning

Simplifying Deep Reinforcement Learning via Self-Supervision

2021-06-10 · Daochen Zha, Kwei-Herng Lai, Kaixiong Zhou, Xia Hu

Supervised regression to demonstrations has been demonstrated to be a stable way to train deep policy networks. We are motivated to study how we can take full advantage of supervised loss functions for stably training de…

Deep Reinforcement Learning regression reinforcement-learning Reinforcement Learning+1

The Utility of Sparse Representations for Control in Reinforcement Learning

2018-11-15 · Vincent Liu, Raksha Kumaraswamy, Lei Le, Martha White

We investigate sparse representations for control in reinforcement learning. While these representations are widely used in computer vision, their prevalence in reinforcement learning is limited to sparse coding where ex…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)