Leveraging the Variance of Return Sequences for Exploration Policy

2020-11-17 · Zerong Xi, Gita Sukthankar

This paper introduces a method for constructing an upper bound for exploration policy using either the weighted variance of return sequences or the weighted temporal difference (TD) error. We demonstrate that the variance of the return sequence for a specific state-action pair is an important information source that can be leveraged to guide exploration in reinforcement learning. The intuition is that fluctuation in the return sequence indicates greater uncertainty in the near future returns. This divergence occurs because of the cyclic nature of value-based reinforcement learning; the evolving value function begets policy improvements which in turn modify the value function. Although both variance and TD errors capture different aspects of this uncertainty, our analysis shows that both can be valuable to guide exploration. We propose a two-stream network architecture to estimate weighted variance/TD errors within DQN agents for our exploration method and show that it outperforms the baseline on a wide range of Atari games.

📄 PDF Abstract BibTeX arXiv:2011.08649

Code (0)

등록된 구현이 없습니다.

Tasks

Atari Games reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Dual-Flow Reinforcement Learning with State-Aware Exploration

2026-06-29 · Qijun Li, Zheng Fu, Qi Song, Yifei He 외 arxiv

In complex continuous-control reinforcement learning tasks, multimodal optimal actions often coincide with uncertain, multimodal return distributions, making reliable value estimation and multimodal exploration challengi…

Reinforcement Learning

Adaptive Exploration for Data-Efficient General Value Function Evaluations

2024-05-13 · Arushi Jain, Josiah P. Hanna, Doina Precup

General Value Functions (GVFs) (Sutton et al., 2011) represent predictive knowledge in reinforcement learning. Each GVF computes the expected return for a given policy, based on a unique reward. Existing methods relying …

MuJoCo

Stein Variational Policy Gradient

2017-04-07 · Yang Liu, Prajit Ramachandran, Qiang Liu, Jian Peng

Policy gradient methods have been successfully applied to many complex reinforcement learning problems. However, policy gradient methods suffer from high variance, slow convergence, and inefficient exploration. In this w…

Bayesian Inference continuous-control Continuous Control Policy Gradient Methods+3

Variance Penalized On-Policy and Off-Policy Actor-Critic

2021-02-03 · Arushi Jain, Gandharv Patil, Ayush Jain, Khimya Khetarpal 외

Reinforcement learning algorithms are typically geared towards optimizing the expected return of an agent. However, in many practical applications, low variance in the return is desired to ensure the reliability of an al…