The Uncertainty Bellman Equation and Exploration
We consider the exploration/exploitation problem in reinforcement learning. For exploitation, it is well known that the Bellman equation connects the value at any time-step to the expected value at subsequent time-steps. In this paper we consider a similar \textit{uncertainty} Bellman equation (UBE), which connects the uncertainty at any time-step to the expected uncertainties at subsequent time-steps, thereby extending the potential exploratory benefit of a policy beyond individual time-steps. We prove that the unique fixed point of the UBE yields an upper bound on the variance of the posterior distribution of the Q-values induced by any policy. This bound can be much tighter than traditional count-based bonuses that compound standard deviation rather than variance. Importantly, and unlike several existing approaches to optimism, this method scales naturally to large systems with complex generalization. Substituting our UBE-exploration strategy for $\epsilon$-greedy improves DQN performance on 51 out of 57 games in the Atari suite.
Code (1)
Tasks
Reinforcement LearningSimilar Papers 제목 키워드 기반
Model-Based Uncertainty in Value Functions
We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. In particular, we focus on characterizing the variance over values induced by a distribution over…
continuous-controlContinuous ControlDeep Reinforcement Learningmodel+5Efficient exploration with Double Uncertain Value Networks
This paper studies directed exploration for reinforcement learning agents by tracking uncertainty about the value of each available action. We identify two sources of uncertainty that are relevant for exploration. The fi…
Efficient ExplorationReinforcement LearningThompson SamplingExponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning
We study risk-sensitive reinforcement learning (RL) based on the entropic risk measure. Although existing works have established non-asymptotic regret guarantees for this problem, they leave open an exponential gap betwe…
reinforcement-learningReinforcement Learning (RL)Valuation of European Options under an Uncertain Market Price of Volatility Risk
We propose a model to quantify the effect of parameter uncertainty on the option price in the Heston model. More precisely, we present a Hamilton-Jacobi-Bellman framework which allows us to evaluate best and worst case s…
Uncertainty QuantificationModel-Bellman Inconsistency for Model-based Offline Reinforcement Learning
For offline reinforcement learning (RL), model-based methods are expected to be data-efficient as they incorporate dynamics models to generate more data. However, due to inevitable model errors, straightforwardly learnin…
D4RLmodelOffline RLreinforcement-learning+3