Actor-Critic Algorithms for Risk-Sensitive MDPs
In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such criteria is known to be a hard problem. In this paper, we consider both discounted and average reward Markov decision processes. For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize. For each of these criteria, we derive a formula for computing its gradient. We then devise actor-critic algorithms for estimating the gradient and updating the policy parameters in the ascent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in a traffic signal control application.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingSequential Decision MakingTraffic Signal ControlSimilar Papers 제목 키워드 기반
Algorithms for CVaR Optimization in MDPs
In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in costs in addition to minimizing a standard criterion. Conditional value-at-risk (CVaR) is a relatively n…
Decision MakingSequential Decision MakingVariance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPs
In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the mos…
Decision MakingSequential Decision MakingTraffic Signal ControlRisk-Constrained Reinforcement Learning with Percentile Risk Criteria
In many sequential decision-making problems one is interested in minimizing an expected cumulative cost while taking into account \emph{risk}, i.e., increased awareness of events of small probability and high consequence…
Decision MakingMarketingreinforcement-learningReinforcement Learning+2Risk-Sensitive RL with Optimized Certainty Equivalents via Reduction to Standard RL
We study Risk-Sensitive Reinforcement Learning (RSRL) with the Optimized Certainty Equivalent (OCE) risk, which generalizes Conditional Value-at-risk (CVaR), entropic risk and Markowitz's mean-variance. Using an augmente…
Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Risk
Robust Markov Decision Processes (RMDPs) have received significant research interest, offering an alternative to standard Markov Decision Processes (MDPs) that often assume fixed transition probabilities. RMDPs address t…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)