An Alternative to Variance: Gini Deviation for Risk-averse Policy Gradient
Restricting the variance of a policy's return is a popular choice in risk-averse Reinforcement Learning (RL) due to its clear mathematical definition and easy interpretability. Traditional methods directly restrict the total return variance. Recent methods restrict the per-step reward variance as a proxy. We thoroughly examine the limitations of these variance-based methods, such as sensitivity to numerical scale and hindering of policy learning, and propose to use an alternative risk measure, Gini deviation, as a substitute. We study various properties of this new risk measure and derive a policy gradient algorithm to minimize it. Empirical evaluation in domains where risk-aversion can be clearly defined, shows that our algorithm can mitigate the limitations of variance-based risk measures and achieves high return with low risk in terms of variance and Gini deviation when others fail to learn a reasonable policy.
Code (0)
등록된 구현이 없습니다.
Tasks
Reinforcement Learning (RL)Similar Papers 제목 키워드 기반
Measures of Variability for Risk-averse Policy Gradient
Risk-averse reinforcement learning (RARL) is critical for decision-making under uncertainty, which is especially valuable in high-stake applications. However, most existing works focus on risk measures, e.g., conditional…
Decision MakingDecision Making Under UncertaintyMean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning
Keeping risk under control is often more crucial than maximizing expected rewards in real-world decision-making situations, such as finance, robotics, autonomous driving, etc. The most natural choice of risk measures is …
Autonomous Drivingcontinuous-controlContinuous ControlDecision Making+4Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning
We present a mean-variance policy iteration (MVPI) framework for risk-averse control in a discounted infinite horizon MDP optimizing the variance of a per-step reward random variable. MVPI enjoys great flexibility in tha…
MuJoCoreinforcement-learningReinforcement LearningReinforcement Learning (RL)A risk measurement approach from risk-averse stochastic optimization of score functions
We propose a risk measurement approach for a risk-averse stochastic problem. We provide results that guarantee that our problem has a solution. We characterize and explore the properties of the argmin as a risk measure a…
regressionStochastic OptimizationOptimal insurance with mean-deviation measures
This paper studies an optimal insurance contracting problem in which the preferences of the decision maker given by the sum of the expected loss and a convex, increasing function of a deviation measure. As for the deviat…