Risk-Averse Action Selection Using Extreme Value Theory Estimates of the CVaR
In a wide variety of sequential decision making problems, it can be important to estimate the impact of rare events in order to minimize risk exposure. A popular risk measure is the conditional value-at-risk (CVaR), which is commonly estimated by averaging observations that occur beyond a quantile at a given confidence level. When this confidence level is very high, this estimation method can exhibit high variance due to the limited number of samples above the corresponding quantile. To mitigate this problem, extreme value theory can be used to derive an estimator for the CVaR that uses extrapolation beyond available samples. This estimator requires the selection of a threshold parameter to work well, which is a difficult challenge that has been widely studied in the extreme value theory literature. In this paper, we present an estimation procedure for the CVaR that combines extreme value theory and a recently introduced method of automated threshold selection by \cite{bader2018automated}. Under appropriate conditions, we estimate the tail risk using a generalized Pareto distribution. We compare empirically this estimation procedure with the commonly used method of sample averaging, and show an improvement in performance for some distributions. We finally show how the estimation procedure can be used in reinforcement learning by applying our method to the multi-arm bandit problem where the goal is to avoid catastrophic risk.
Code (1)
Tasks
Decision MakingReinforcement LearningSequential Decision MakingSimilar Papers 제목 키워드 기반
Extreme Risk Mitigation in Reinforcement Learning using Extreme Value Theory
Risk-sensitive reinforcement learning (RL) has garnered significant attention in recent years due to the growing interest in deploying RL agents in real-world scenarios. A critical aspect of risk awareness involves model…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Risk-Averse Reinforcement Learning via Dynamic Time-Consistent Risk Measures
Traditional reinforcement learning (RL) aims to maximize the expected total reward, while the risk of uncertain outcomes needs to be controlled to ensure reliable performance in a risk-averse setting. In this paper, we c…
Q-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)A Zeroth-Order Momentum Method for Risk-Averse Online Convex Games
We consider risk-averse learning in repeated unknown games where the goal of the agents is to minimize their individual risk of incurring significantly high cost. Specifically, the agents use the conditional value at ris…
Risk averse non-stationary multi-armed bandits
This paper tackles the risk averse multi-armed bandits problem when incurred losses are non-stationary. The conditional value-at-risk (CVaR) is used as the objective function. Two estimation methods are proposed for this…
Multi-Armed BanditsOption Hedging with Risk Averse Reinforcement Learning
In this paper we show how risk-averse reinforcement learning can be used to hedge options. We apply a state-of-the-art risk-averse algorithm: Trust Region Volatility Optimization (TRVO) to a vanilla option hedging enviro…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)