Policy Gradient Methods for Risk-Sensitive Distributional Reinforcement Learning with Provable Convergence

2024-05-23 · Minheng Xiao, Xian Yu, Lei Ying

Risk-sensitive reinforcement learning (RL) is crucial for maintaining reliable performance in high-stakes applications. While traditional RL methods aim to learn a point estimate of the random cumulative cost, distributional RL (DRL) seeks to estimate the entire distribution of it, which leads to a unified framework for handling different risk measures. However, developing policy gradient methods for risk-sensitive DRL is inherently more complex as it involves finding the gradient of a probability measure. This paper introduces a new policy gradient method for risk-sensitive DRL with general coherent risk measures, where we provide an analytical form of the probability measure's gradient for any distribution. For practical use, we design a categorical distributional policy gradient algorithm (CDPG) that approximates any distribution by a categorical family supported on some fixed points. We further provide a finite-support optimality guarantee and a finite-iteration convergence guarantee under inexact policy evaluation and gradient estimation. Through experiments on stochastic Cliffwalk and CartPole environments, we illustrate the benefits of considering a risk-sensitive setting in DRL.

📄 PDF Abstract BibTeX arXiv:2405.14749

Code (0)

등록된 구현이 없습니다.

Tasks

Distributional Reinforcement LearningPolicy Gradient MethodsReinforcement Learning (RL)

RS-Diffuser: Risk-Sensitive Diffusion Planning with Distributional Value Guidance

2026-06-26 · Shiqiang Gong arxiv

Offline reinforcement learning enables policy learning from fixed datasets without additional environment interaction, making it appealing for safety-critical applications where online exploration is costly or unsafe. Di…

Reinforcement Learning Robot Navigation Offline RL

Distributional Reinforcement Learning for Risk-Sensitive Policies

2021-01-01 · Shiau Hong Lim, Ilyas Malik

We address the problem of learning a risk-sensitive policy based on the CVaR risk measure using distributional reinforcement learning. In particular, we show that applying the distributional Bellman optimality operator w…

Distributional Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

A Risk-Sensitive Approach to Policy Optimization

2022-08-19 · Jared Markowitz, Ryan W. Gardner, Ashley Llorens, Raman Arora 외

Standard deep reinforcement learning (DRL) aims to maximize expected reward, considering collected experiences equally in formulating a policy. This differs from human decision-making, where gains and losses are valued d…

Decision Making Deep Reinforcement Learning

Cautious Reinforcement Learning via Distributional Risk in the Dual Domain

2020-02-27 · Junyu Zhang, Amrit Singh Bedi, Mengdi Wang, Alec Koppel

We study the estimation of risk-sensitive policies in reinforcement learning problems defined by a Markov Decision Process (MDPs) whose state and action spaces are countably finite. Prior efforts are predominately afflic…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Risk-Sensitive Exponential Actor Critic

2026-02-06 · Alonso Granados, Jason Pacheco arxiv

Model-free deep reinforcement learning (RL) algorithms have achieved tremendous success on a range of challenging tasks. However, safety concerns remain when these methods are deployed on real-world applications, necessi…

Reinforcement Learning