Entropic Risk-Sensitive Reinforcement Learning: A Meta Regret Framework with Function Approximation
We study risk-sensitive reinforcement learning with the entropic risk measure and function approximation. We consider the finite-horizon episodic MDP setting, and propose a meta algorithm based on value iteration. We then derive two algorithms for linear and general function approximation, namely RSVI.L and RSVI.G, respectively, as special instances of the meta algorithm. We illustrate that the success of RSVI.L depends crucially on carefully designed feature mapping and regularization that adapt to risk sensitivity. In addition, both RSVI.L and RSVI.G maintain risk-sensitive optimism that facilitates efficient exploration. On the analytic side, we provide regret analysis for the algorithms by developing a meta analytic framework, at the core of which is a risk-sensitive optimism condition. We show that any instance of the meta algorithm that satisfies the condition yields a meta regret bound. We further verify the condition for RSVI.L and RSVI.G under respective function approximation settings to obtain concrete regret bounds that scale sublinearly in the number of episodes.
Code (0)
등록된 구현이 없습니다.
Tasks
Efficient Explorationreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Risk-Sensitive RL with Optimized Certainty Equivalents via Reduction to Standard RL
We study Risk-Sensitive Reinforcement Learning (RSRL) with the Optimized Certainty Equivalent (OCE) risk, which generalizes Conditional Value-at-risk (CVaR), entropic risk and Markowitz's mean-variance. Using an augmente…
Taming Equilibrium Bias in Risk-Sensitive Multi-Agent Reinforcement Learning
We study risk-sensitive multi-agent reinforcement learning under general-sum Markov games, where agents optimize the entropic risk measure of rewards with possibly diverse risk preferences. We show that using the regret …
Multi-agent Reinforcement Learningreinforcement-learningNon-stationary Risk-sensitive Reinforcement Learning: Near-optimal Dynamic Regret, Adaptive Detection, and Separation Design
We study risk-sensitive reinforcement learning (RL) based on an entropic risk measure in episodic non-stationary Markov decision processes (MDPs). Both the reward functions and the state transition kernels are unknown an…
Reinforcement Learning (RL)Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning
We study risk-sensitive reinforcement learning (RL) based on the entropic risk measure. Although existing works have established non-asymptotic regret guarantees for this problem, they leave open an exponential gap betwe…
reinforcement-learningReinforcement Learning (RL)Cascaded Gaps: Towards Gap-Dependent Regret for Risk-Sensitive Reinforcement Learning
In this paper, we study gap-dependent regret guarantees for risk-sensitive reinforcement learning based on the entropic risk measure. We propose a novel definition of sub-optimality gaps, which we call cascaded gaps, and…
reinforcement-learningReinforcement Learning (RL)