Rethinking Value Function Learning for Generalization in Reinforcement Learning

2022-10-18 · Seungyong Moon, JunYeong Lee, Hyun Oh Song

Our work focuses on training RL agents on multiple visually diverse environments to improve observational generalization performance. In prior methods, policy and value networks are separately optimized using a disjoint network architecture to avoid interference and obtain a more accurate value function. We identify that a value network in the multi-environment setting is more challenging to optimize and prone to memorizing the training data than in the conventional single-environment setting. In addition, we find that appropriate regularization on the value network is necessary to improve both training and test performance. To this end, we propose Delayed-Critic Policy Gradient (DCPG), a policy gradient algorithm that implicitly penalizes value estimates by optimizing the value network less frequently with more training data than the policy network. This can be implemented using a single unified network architecture. Furthermore, we introduce a simple self-supervised task that learns the forward and inverse dynamics of environments using a single discriminator, which can be jointly optimized with the value network. Our proposed algorithms significantly improve observational generalization performance and sample efficiency on the Procgen Benchmark.

📄 PDF Abstract BibTeX arXiv:2210.09960

Code (1)

snu-mllab/dcpg 공식 구현 pytorch

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Test 설명 없음

Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach

2019-02-08 · Silviu Pitis

Reinforcement learning (RL) agents have traditionally been tasked with maximizing the value function of a Markov decision process (MDP), either in continuous settings, with fixed discount factor $\gamma < 1$, or in episo…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)+1

Hierarchical Universal Value Function Approximators

2024-10-11 · Rushiv Arora

There have been key advancements to building universal approximators for multi-goal collections of reinforcement learning value functions -- key elements in estimating long-term returns of states in a parameterized manne…

Hierarchical Reinforcement Learning reinforcement-learning Reinforcement Learning

Generalization and Exploration via Randomized Value Functions

2014-02-04 · Ian Osband, Benjamin Van Roy, Zheng Wen

We propose randomized least-squares value iteration (RLSVI) -- a new reinforcement learning algorithm designed to explore and generalize efficiently via linearly parameterized value functions. We explain why versions of …

Efficient Exploration reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Efficient Reinforcement Learning in Deterministic Systems with Value Function Generalization

2013-07-18 · Zheng Wen, Benjamin Van Roy

We consider the problem of reinforcement learning over episodes of a finite-horizon deterministic system and as a solution propose optimistic constraint propagation (OCP), an algorithm designed to synthesize efficient ex…

Efficient Exploration reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Decoupling Value and Policy for Generalization in Reinforcement Learning

2021-02-20 · Roberta Raileanu, Rob Fergus

Standard deep reinforcement learning algorithms use a shared representation for the policy and value function, especially when training directly from images. However, we argue that more information is needed to accuratel…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)