Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization
We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. In particular, we focus on characterizing the variance over values induced by a distribution over Markov decision processes (MDPs). Previous work upper bounds the posterior variance over values by solving a so-called uncertainty Bellman equation (UBE), but the over-approximation may result in inefficient exploration. We propose a new UBE whose solution converges to the true posterior variance over values and leads to lower regret in tabular exploration problems. We identify challenges to apply the UBE theory beyond tabular problems and propose a suitable approximation. Based on this approximation, we introduce a general-purpose policy optimization algorithm, Q-Uncertainty Soft Actor-Critic (QU-SAC), that can be applied for either risk-seeking or risk-averse policy optimization with minimal changes. Experiments in both online and offline RL demonstrate improved performance compared to other uncertainty estimation methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Model-based Reinforcement LearningOffline RLMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Intrinsic Randomness in Epidemic Modelling Beyond Statistical Uncertainty
Uncertainty can be classified as either aleatoric (intrinsic randomness) or epistemic (imperfect knowledge of parameters). The majority of frameworks assessing infectious disease risk consider only epistemic uncertainty.…
Variational Bayesian Reinforcement Learning with Regret Bounds
In reinforcement learning the Q-values summarize the expected future rewards that the agent will attain. However, they cannot capture the epistemic uncertainty about those rewards. In this work we derive a new Bellman op…
Q-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Epistemic Risk-Sensitive Reinforcement Learning
We develop a framework for interacting with uncertain environments in reinforcement learning (RL) by leveraging preferences in the form of utility functions. We claim that there is value in considering different risk mea…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Uncertainty-aware Distributional Offline Reinforcement Learning
Offline reinforcement learning (RL) presents distinct challenges as it relies solely on observational data. A central concern in this context is ensuring the safety of the learned policy by quantifying uncertainties asso…
Offline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Distributional Reinforcement Learning with Online Risk-awareness Adaption
The use of reinforcement learning (RL) in practical applications requires considering sub-optimal outcomes, which depend on the agent's familiarity with the uncertain environment. Dynamically adjusting the level of epist…
Distributional Reinforcement Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)