A policy gradient approach for optimization of smooth risk measures
We propose policy gradient algorithms for solving a risk-sensitive reinforcement learning (RL) problem in on-policy as well as off-policy settings. We consider episodic Markov decision processes, and model the risk using the broad class of smooth risk measures of the cumulative discounted reward. We propose two template policy gradient algorithms that optimize a smooth risk measure in on-policy and off-policy RL settings, respectively. We derive non-asymptotic bounds that quantify the rate of convergence of our proposed algorithms to a stationary point of the smooth risk measure. As special cases, we establish that our algorithms apply to optimization of mean-variance and distortion risk measures, respectively.
Code (0)
등록된 구현이 없습니다.
Tasks
reinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Risk-sensitive reinforcement learning using expectiles, shortfall risk and optimized certainty equivalent risk
We propose risk-sensitive reinforcement learning algorithms catering to three families of risk measures, namely expectiles, utility-based shortfall risk and optimized certainty equivalent risk. For each risk measure, in …
Reinforcement LearningPolicy Gradient for Coherent Risk Measures
Several authors have recently developed risk-sensitive policy gradient methods that augment the standard expected cost minimization problem with a measure of variability in cost. These studies have focused on specific ri…
Policy Gradient MethodsReinforcement LearningRisk-Sensitive Reinforcement Learning via Policy Gradient Search
The objective in a traditional reinforcement learning (RL) problem is to find a policy that optimizes the expected value of a performance metric such as the infinite-horizon cumulative discounted or long-run average cost…
Policy Gradient Methodsreinforcement-learningReinforcement LearningReinforcement Learning (RL)Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problems
Stochastic optimization has found wide applications in minimizing objective functions in machine learning, which motivates a lot of theoretical studies to understand its practical success. Most of existing studies focus …
Stochastic OptimizationReinforcement Learning with Dynamic Convex Risk Measures
We develop an approach for solving time-consistent risk-sensitive stochastic optimization problems using model-free reinforcement learning (RL). Specifically, we assume agents assess the risk of a sequence of random vari…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Stochastic Optimization