Robust Bayesian Dynamic Programming for On-policy Risk-sensitive Reinforcement Learning

2025-12-31 · Shanyu Han, Yangbo He, Yang Liu arxiv

We propose a novel framework for risk-sensitive reinforcement learning (RSRL) that incorporates robustness against transition uncertainty. We define two distinct yet coupled risk measures: an inner risk measure addressing state and cost randomness and an outer risk measure capturing transition dynamics uncertainty. Our framework unifies and generalizes most existing RL frameworks by permitting general coherent risk measures for both inner and outer risk measures. Within this framework, we construct a risk-sensitive robust Markov decision process (RSRMDP), derive its Bellman equation, and provide error analysis under a given posterior distribution. We further develop a Bayesian Dynamic Programming (Bayesian DP) algorithm that alternates between posterior updates and value iteration. The approach employs an estimator for the risk-based Bellman operator that combines Monte Carlo sampling with convex optimization, for which we prove strong consistency guarantees. Furthermore, we demonstrate that the algorithm converges to a near-optimal policy in the training environment and analyze both the sample complexity and the computational complexity under the Dirichlet posterior and CVaR. Finally, we validate our approach through two numerical experiments. The results exhibit excellent convergence properties while providing intuitive demonstrations of its advantages in both risk-sensitivity and robustness. Empirically, we further demonstrate the advantages of the proposed algorithm through an application on option hedging.

📄 PDF Abstract BibTeX arXiv:2512.24580

Code (0)

등록된 구현이 없습니다.

Tasks

Reinforcement Learning

On the Convergence of Modified Policy Iteration in Risk Sensitive Exponential Cost Markov Decision Processes

2023-02-08 · Yashaswini Murthy, Mehrdad Moharrami, R. Srikant

Modified policy iteration (MPI) is a dynamic programming algorithm that combines elements of policy iteration and value iteration. The convergence of MPI has been well studied in the context of discounted and average-cos…

Computational Efficiency

Bayesian Risk-Sensitive Policy Optimization For MDPs With General Loss Functions

2025-09-19 · Xiaoshuang Wang, Yifan Lin, Enlu Zhou arxiv

Motivated by many application problems, we consider Markov decision processes (MDPs) with a general loss function and unknown parameters. To mitigate the epistemic uncertainty associated with unknown parameters, we take …

Policy Gradient for Coherent Risk Measures

2015-02-13 · NeurIPS 2015 12 · Aviv Tamar, Yin-Lam Chow, Mohammad Ghavamzadeh, Shie Mannor

Several authors have recently developed risk-sensitive policy gradient methods that augment the standard expected cost minimization problem with a measure of variability in cost. These studies have focused on specific ri…

Policy Gradient Methods Reinforcement Learning

Reinforcement Learning with Dynamic Convex Risk Measures

2021-12-26 · Anthony Coache, Sebastian Jaimungal

We develop an approach for solving time-consistent risk-sensitive stochastic optimization problems using model-free reinforcement learning (RL). Specifically, we assume agents assess the risk of a sequence of random vari…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Stochastic Optimization

Epistemic Risk-Sensitive Reinforcement Learning

2019-06-14 · Hannes Eriksson, Christos Dimitrakakis

We develop a framework for interacting with uncertain environments in reinforcement learning (RL) by leveraging preferences in the form of utility functions. We claim that there is value in considering different risk mea…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)