Uncertainty-sensitive Learning and Planning with Ensembles
We propose a reinforcement learning framework for discrete environments in which an agent makes both strategic and tactical decisions. The former manifests itself through the use of value function, while the latter is powered by a tree search planner. These tools complement each other. The planning module performs a local \textit{what-if} analysis, which allows to avoid tactical pitfalls and boost backups of the value function. The value function, being global in nature, compensates for inherent locality of the planner. In order to further solidify this synergy, we introduce an exploration mechanism with two distinctive components: uncertainty modelling and risk measurement. To model the uncertainty we use value function ensembles, and to reflect risk we use propose several functionals that summarize the implied by the ensemble. We show that our method performs well on hard exploration environments: Deep-sea, toy Montezuma's Revenge, and Sokoban. In all the cases, we obtain speed-up in learning and boost in performance.
Code (1)
Tasks
Montezuma's RevengeReinforcement LearningSokobanSimilar Papers 제목 키워드 기반
Uncertainty - sensitive learning and planning with ensembles
We propose a reinforcement learning framework for discrete environments in which an agent optimizes its behavior on two timescales. For the short one, it uses tree search methods to perform tactical decisions. The long s…
Montezuma's RevengeSokobanDeep Ensembles to Improve Uncertainty Quantification of Statistical Downscaling Models under Climate Change Conditions
Recently, deep learning has emerged as a promising tool for statistical downscaling, the set of methods for generating high-resolution climate fields from coarse low-resolution variables. Nevertheless, their ability to g…
Uncertainty QuantificationEpistemic Robust Offline Reinforcement Learning
Offline reinforcement learning learns policies from fixed datasets without further environment interaction. A key challenge in this setting is epistemic uncertainty, arising from limited or biased data coverage, particul…
Reinforcement LearningOffline RLRobust and Adaptive Planning under Model Uncertainty
Planning under model uncertainty is a fundamental problem across many applications of decision making and learning. In this paper, we propose the Robust Adaptive Monte Carlo Planning (RAMCP) algorithm, which allows compu…
Computational EfficiencyDecision MakingmodelDR-PETS: Learning-Based Control With Planning in Adversarial Environments
Ensuring robustness against epistemic, possibly adversarial, perturbations is essential for reliable real-world decision-making. While the Probabilistic Ensembles with Trajectory Sampling (PETS) algorithm inherently hand…
Decision Making