Risk-Sensitive Markov Decision Processes with Combined Metrics of Mean and Variance

2020-08-09 · Li Xia

This paper investigates the optimization problem of an infinite stage discrete time Markov decision process (MDP) with a long-run average metric considering both mean and variance of rewards together. Such performance metric is important since the mean indicates average returns and the variance indicates risk or fairness. However, the variance metric couples the rewards at all stages, the traditional dynamic programming is inapplicable as the principle of time consistency fails. We study this problem from a new perspective called the sensitivity-based optimization theory. A performance difference formula is derived and it can quantify the difference of the mean-variance combined metrics of MDPs under any two different policies. The difference formula can be utilized to generate new policies with strictly improved mean-variance performance. A necessary condition of the optimal policy and the optimality of deterministic policies are derived. We further develop an iterative algorithm with a form of policy iteration, which is proved to converge to local optima both in the mixed and randomized policy space. Specially, when the mean reward is constant in policies, the algorithm is guaranteed to converge to the global optimum. Finally, we apply our approach to study the fluctuation reduction of wind power in an energy storage system, which demonstrates the potential applicability of our optimization method.

📄 PDF Abstract BibTeX arXiv:2008.03707

Code (0)

등록된 구현이 없습니다.

Tasks

Fairness

Markov Decision Processes with Risk-Sensitive Criteria: An Overview

2023-11-12 · Nicole Bäuerle, Anna Jaśkiewicz

The paper provides an overview of the theory and applications of risk-sensitive Markov decision processes. The term 'risk-sensitive' refers here to the use of the Optimized Certainty Equivalent as a means to measure expe…

An Actor-Critic Algorithm with Function Approximation for Risk Sensitive Cost Markov Decision Processes

2025-02-17 · Soumyajit Guin, Vivek S. Borkar, Shalabh Bhatnagar

In this paper, we consider the risk-sensitive cost criterion with exponentiated costs for Markov decision processes and develop a model-free policy gradient algorithm in this setting. Unlike additive cost criteria such a…

Verification of Markov Decision Processes with Risk-Sensitive Measures

2018-02-28 · Murat Cubuktepe, Ufuk Topcu

We develop a method for computing policies in Markov decision processes with risk-sensitive measures subject to temporal logic constraints. Specifically, we use a particular risk-sensitive measure from cumulative prospec…

Inverse Risk-Sensitive Reinforcement Learning

2017-03-29 · Lillian J. Ratliff, Eric Mazumdar

We address the problem of inverse reinforcement learning in Markov decision processes where the agent is risk-sensitive. In particular, we model risk-sensitivity in a reinforcement learning framework by making use of mod…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Robust Risk-Sensitive Reinforcement Learning with Conditional Value-at-Risk

2024-05-02 · Xinyi Ni, Lifeng Lai

Robust Markov Decision Processes (RMDPs) have received significant research interest, offering an alternative to standard Markov Decision Processes (MDPs) that often assume fixed transition probabilities. RMDPs address t…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)