Exploration via Epistemic Value Estimation

2023-03-07 · Simon Schmitt, John Shawe-Taylor, Hado van Hasselt

How to efficiently explore in reinforcement learning is an open problem. Many exploration algorithms employ the epistemic uncertainty of their own value predictions -- for instance to compute an exploration bonus or upper confidence bound. Unfortunately the required uncertainty is difficult to estimate in general with function approximation. We propose epistemic value estimation (EVE): a recipe that is compatible with sequential decision making and with neural network function approximators. It equips agents with a tractable posterior over all their parameters from which epistemic value uncertainty can be computed efficiently. We use the recipe to derive an epistemic Q-Learning agent and observe competitive performance on a series of benchmarks. Experiments confirm that the EVE recipe facilitates efficient exploration in hard exploration tasks.

📄 PDF Abstract BibTeX arXiv:2303.04012

Code (0)

등록된 구현이 없습니다.

Tasks

Decision MakingEfficient ExplorationQ-LearningSequential Decision Making

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

A unified uncertainty-aware exploration: Combining epistemic and aleatory uncertainty

2024-01-05 · Parvin Malekzadeh, Ming Hou, Konstantinos N. Plataniotis

Exploration is a significant challenge in practical reinforcement learning (RL), and uncertainty-aware exploration that incorporates the quantification of epistemic and aleatory uncertainty has been recognized as an effe…

Decision Making Reinforcement Learning (RL)

Epistemic Monte Carlo Tree Search

2022-10-21 · Yaniv Oren, Villiam Vadocz, Matthijs T. J. Spaan, Wendelin Böhmer

The AlphaZero/MuZero (A/MZ) family of algorithms has achieved remarkable success across various challenging domains by integrating Monte Carlo Tree Search (MCTS) with learned models. Learned models introduce epistemic un…

Model-based Reinforcement Learning reinforcement-learning Reinforcement Learning (RL)

Wasserstein Actor-Critic: Directed Exploration via Optimism for Continuous-Actions Control

2023-03-04 · Amarildo Likmeta, Matteo Sacco, Alberto Maria Metelli, Marcello Restelli

Uncertainty quantification has been extensively used as a means to achieve efficient directed exploration in Reinforcement Learning (RL). However, state-of-the-art methods for continuous actions still suffer from high sa…

MuJoCo Q-Learning Reinforcement Learning (RL)Uncertainty Quantification

Evolving Robustness--Exploration Trade-off in Online Reinforcement Learning via Quantile Bayesian Risk MDPs

2026-05-23 · Meichen Song, Yuhao Wang, Enlu Zhou arxiv

In online reinforcement learning, data scarcity creates epistemic uncertainty that makes robustness important early in learning, whereas sufficient exploration is needed to learn the true-environment optimal policy. We s…

Reinforcement Learning

Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization

2023-12-07 · Carlos E. Luis, Alessandro G. Bottero, Julia Vinogradska, Felix Berkenkamp 외

We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. In particular, we focus on characterizing the variance over values induced by a distribution over…

Model-based Reinforcement Learning Offline RL