Distributional Reinforcement Learning for Risk-Sensitive Policies

2021-01-01 · Shiau Hong Lim, Ilyas Malik

We address the problem of learning a risk-sensitive policy based on the CVaR risk measure using distributional reinforcement learning. In particular, we show that applying the distributional Bellman optimality operator with respect to a risk-based action-selection strategy overestimates the dynamic, Markovian CVaR. The resulting policies can however still be overly conservative and one often prefers to learn an optimal policy based on the static, non-Markovian CVaR. To this end, we propose a modification to the existing algorithm and show that it can indeed learn a proper CVaR-optimized policy. Our proposed approach is a simple extension of standard distributional RL algorithms and can therefore take advantage of many of the recent advances in deep RL. On both synthetic and real data, we empirically show that our proposed algorithm is able to produce a family of risk-averse policies that achieves a better tradeoff between risk and the expected return.

📄 PDF Abstract BibTeX

Code (0)

등록된 구현이 없습니다.

Tasks

Distributional Reinforcement Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Implicit Quantile Networks for Distributional Reinforcement Learning

2018-06-14 · ICML 2018 7 · Will Dabney, Georg Ostrovski, David Silver, Rémi Munos

In this work, we build on recent advances in distributional reinforcement learning to give a generally applicable, flexible, and state-of-the-art distributional variant of DQN. We achieve this by using quantile regressio…

Atari Games Distributional Reinforcement Learning quantile regression regression+3

Is Risk-Sensitive Reinforcement Learning Properly Resolved?

2023-07-02 · Ruiwen Zhou, Minghuan Liu, Kan Ren, Xufang Luo 외

Due to the nature of risk management in learning applicable policies, risk-sensitive reinforcement learning (RSRL) has been realized as an important direction. RSRL is usually achieved by learning risk-sensitive objectiv…

Distributional Reinforcement Learning Management Q-Learning reinforcement-learning+1

Risk-Sensitive Policy with Distributional Reinforcement Learning

2022-12-30 · Thibaut Théate, Damien Ernst

Classical reinforcement learning (RL) techniques are generally concerned with the design of decision-making policies driven by the maximisation of the expected outcome. Nevertheless, this approach does not take into cons…

Decision Making Distributional Reinforcement Learning reinforcement-learning Reinforcement Learning+2

RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization

2023-11-03 · NeurIPS 2023 11 · Siqi Shen, Chennan Ma, Chao Li, Weiquan Liu 외

Multi-agent systems are characterized by environmental uncertainty, varying policies of agents, and partial observability, which result in significant risks. In the context of Multi-Agent Reinforcement Learning (MARL), l…

Multi-agent Reinforcement Learning reinforcement-learning Reinforcement Learning

Risk-averse policies for natural gas futures trading using distributional reinforcement learning

2025-01-08 · Félicien Hêche, Biagio Nigro, Oussama Barakat, Stephan Robert-Nicoud

Financial markets have experienced significant instabilities in recent years, creating unique challenges for trading and increasing interest in risk-averse strategies. Distributional Reinforcement Learning (RL) algorithm…

Distributional Reinforcement Learning energy trading quantile regression Reinforcement Learning (RL)