Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexity

2023-06-20 · Runyu Zhang, Yang Hu, Na Li

Robust Markov Decision Processes (MDPs) and risk-sensitive MDPs are both powerful tools for making decisions in the presence of uncertainties. Previous efforts have aimed to establish their connections, revealing equivalences in specific formulations. This paper introduces a new formulation for risk-sensitive MDPs, which assesses risk in a slightly different manner compared to the classical Markov risk measure (Ruszczy\'nski 2010), and establishes its equivalence with a class of soft robust MDP (RMDP) problems, including the standard RMDP as a special case. Leveraging this equivalence, we further derive the policy gradient theorem for both problems, proving gradient domination and global convergence of the exact policy gradient method under the tabular setting with direct parameterization. This forms a sharp contrast to the Markov risk measure, known to be potentially non-gradient-dominant (Huang et al. 2021). We also propose a sample-based offline learning algorithm, namely the robust fitted-Z iteration (RFZI), for a specific soft RMDP problem with a KL-divergence regularization term (or equivalently the risk-sensitive MDP with an entropy risk measure). We showcase its streamlined design and less stringent assumptions due to the equivalence and analyze its sample complexity

📄 PDF Abstract BibTeX arXiv:2306.11626

Code (1)

huyangsh/risk-sensitive-RL_ICRL-2024 공식 구현 pytorch

Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Risk

2024-05-02 · Xinyi Ni, Lifeng Lai

Robust Markov Decision Processes (RMDPs) have received significant research interest, offering an alternative to standard Markov Decision Processes (MDPs) that often assume fixed transition probabilities. RMDPs address t…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Utility-Constrained Policy Optimization

2026-06-12 · Mehrdad Moghimi, Bernardo Avila Pires arxiv

Constrained MDPs (CMDPs) are a widely adopted framework for incorporating safety into RL agents; however, the framework does not support risk-sensitive constraints. This can be problematic: For example, CMDPs allow for o…

Efficient Policy Iteration for Robust Markov Decision Processes via Regularization

2022-05-28 · Navdeep Kumar, Kfir Levy, Kaixin Wang, Shie Mannor

Robust Markov decision processes (MDPs) provide a general framework to model decision problems where the system dynamics are changing or only partially known. Efficient methods for some \texttt{sa}-rectangular robust MDP…

Provably Efficient CVaR RL in Low-rank MDPs

2023-11-20 · Yulai Zhao, Wenhao Zhan, Xiaoyan Hu, Ho-fung Leung 외

We study risk-sensitive Reinforcement Learning (RL), where we aim to maximize the Conditional Value at Risk (CVaR) with a fixed risk tolerance $\tau$. Prior theoretical work studying risk-sensitive RL focuses on the tabu…

Reinforcement Learning (RL)Representation Learning

Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach

2015-06-06 · NeurIPS 2015 12 · Yin-Lam Chow, Aviv Tamar, Shie Mannor, Marco Pavone

In this paper we address the problem of decision making within a Markov decision process (MDP) framework where risk and modeling errors are taken into account. Our approach is to minimize a risk-sensitive conditional-val…

Decision Making