SENTINEL: Taming Uncertainty with Ensemble-based Distributional Reinforcement Learning

2021-02-22 · Hannes Eriksson, Debabrota Basu, Mina Alibeigi, Christos Dimitrakakis

In this paper, we consider risk-sensitive sequential decision-making in Reinforcement Learning (RL). Our contributions are two-fold. First, we introduce a novel and coherent quantification of risk, namely composite risk, which quantifies the joint effect of aleatory and epistemic risk during the learning process. Existing works considered either aleatory or epistemic risk individually, or as an additive combination. We prove that the additive formulation is a particular case of the composite risk when the epistemic risk measure is replaced with expectation. Thus, the composite risk is more sensitive to both aleatory and epistemic uncertainty than the individual and additive formulations. We also propose an algorithm, SENTINEL-K, based on ensemble bootstrapping and distributional RL for representing epistemic and aleatory uncertainty respectively. The ensemble of K learners uses Follow The Regularised Leader (FTRL) to aggregate the return distributions and obtain the composite risk. We experimentally verify that SENTINEL-K estimates the return distribution better, and while used with composite risk estimates, demonstrates higher risk-sensitive performance than state-of-the-art risk-sensitive and distributional RL algorithms.

📄 PDF Abstract BibTeX arXiv:2102.11075

Code (0)

등록된 구현이 없습니다.

Tasks

Decision MakingDistributional Reinforcement LearningModel-based Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Sequential Decision Making

Diverse Projection Ensembles for Distributional Reinforcement Learning

2023-06-12 · Moritz A. Zanger, Wendelin Böhmer, Matthijs T. J. Spaan

In contrast to classical reinforcement learning (RL), distributional RL algorithms aim to learn the distribution of returns rather than their expected value. Since the nature of the return distribution is generally unkno…

Distributional Reinforcement Learning Diversity Inductive Bias reinforcement-learning+2

Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving

2021-05-21 · Carl-Johan Hoel, Krister Wolff, Leo Laine

Reinforcement learning (RL) can be used to create a decision-making agent for autonomous driving. However, previous approaches provide only black-box solutions, which do not offer information on how confident the agent i…

Autonomous Driving Decision Making reinforcement-learning Reinforcement Learning (RL)

Distributional Reinforcement Learning with Ensembles

2020-03-24 · Björn Lindenberg, Jonas Nordqvist, Karl-Olof Lindahl

It is well known that ensemble methods often provide enhanced performance in reinforcement learning. In this paper, we explore this concept further by using group-aided training within the distributional reinforcement le…

Distributional Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control

2022-07-27 · Takuya Kanazawa, HaiYan Wang, Chetan Gupta

Uncertainty quantification is one of the central challenges for machine learning in real-world applications. In reinforcement learning, an agent confronts two kinds of uncertainty, called epistemic uncertainty and aleato…

continuous-control Continuous Control reinforcement-learning Reinforcement Learning+2

A Unifying Lens on Reward Uncertainty in RLHF

2026-06-08 · Ely Hahami, Yoel Zimmermann, Ray Zhou, Jack Benarroch Jedlicki arxiv

Reinforcement learning from human feedback (RLHF) is bottlenecked by reward hacking, where the policy exploits errors in a proxy reward model (RM) and produces high RM scores without genuine quality gains. A natural miti…

Reinforcement Learning Bayesian Inference