Mean-Semivariance Policy Optimization via Risk-Averse Reinforcement Learning

2022-06-15 · Xiaoteng Ma, Shuai Ma, Li Xia, Qianchuan Zhao

Keeping risk under control is often more crucial than maximizing expected rewards in real-world decision-making situations, such as finance, robotics, autonomous driving, etc. The most natural choice of risk measures is variance, which penalizes the upside volatility as much as the downside part. Instead, the (downside) semivariance, which captures the negative deviation of a random variable under its mean, is more suitable for risk-averse proposes. This paper aims at optimizing the mean-semivariance (MSV) criterion in reinforcement learning w.r.t. steady reward distribution. Since semivariance is time-inconsistent and does not satisfy the standard Bellman equation, the traditional dynamic programming methods are inapplicable to MSV problems directly. To tackle this challenge, we resort to Perturbation Analysis (PA) theory and establish the performance difference formula for MSV. We reveal that the MSV problem can be solved by iteratively solving a sequence of RL problems with a policy-dependent reward function. Further, we propose two on-policy algorithms based on the policy gradient theory and the trust region method. Finally, we conduct diverse experiments from simple bandit problems to continuous control tasks in MuJoCo, which demonstrate the effectiveness of our proposed methods.

📄 PDF Abstract BibTeX arXiv:2206.07376

Code (0)

등록된 구현이 없습니다.

Tasks

Autonomous Driving continuous-control Continuous Control Decision MakingMuJoCoreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Mean-Variance Policy Iteration for Risk-Averse Reinforcement Learning

2020-04-22 · Shangtong Zhang, Bo Liu, Shimon Whiteson

We present a mean-variance policy iteration (MVPI) framework for risk-averse control in a discounted infinite horizon MDP optimizing the variance of a per-step reward random variable. MVPI enjoys great flexibility in tha…

MuJoCo reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

STOPS: Short-Term-based Volatility-controlled Policy Search and its Global Convergence

2022-01-24 · Liangliang Xu, Daoming Lyu, Yangchen Pan, Aiwen Jiang 외

It remains challenging to deploy existing risk-averse approaches to real-world applications. The reasons are multi-fold, including the lack of global optimality guarantee and the necessity of learning from long-term cons…

MuJoCo

Measures of Variability for Risk-averse Policy Gradient

2025-04-15 · Yudong Luo, Yangchen Pan, Jiaqi Tan, Pascal Poupart

Risk-averse reinforcement learning (RARL) is critical for decision-making under uncertainty, which is especially valuable in high-stake applications. However, most existing works focus on risk measures, e.g., conditional…

Decision Making Decision Making Under Uncertainty

Efficient Risk-Averse Reinforcement Learning

2022-05-10 · Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, Shie Mannor

In risk-averse reinforcement learning (RL), the goal is to optimize some risk measure of the returns. A risk measure often focuses on the worst returns out of the agent's experience. As a result, standard methods for ris…

Autonomous Driving reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Policy Gradient Bayesian Robust Optimization for Imitation Learning

2021-06-11 · Zaynah Javed, Daniel S. Brown, Satvik Sharma, Jerry Zhu 외

The difficulty in specifying rewards for many real-world problems has led to an increased focus on learning rewards from human feedback, such as demonstrations. However, there are often many different reward functions th…

Imitation Learning