Quantile Markov Decision Process
The goal of a traditional Markov decision process (MDP) is to maximize expected cumulativereward over a defined horizon (possibly infinite). In many applications, however, a decision maker may beinterested in optimizing a specific quantile of the cumulative reward instead of its expectation. In this paperwe consider the problem of optimizing the quantiles of the cumulative rewards of a Markov decision process(MDP), which we refer to as a quantile Markov decision process (QMDP). We provide analytical resultscharacterizing the optimal QMDP value function and present a dynamic programming-based algorithm tosolve for the optimal policy. The algorithm also extends to the MDP problem with a conditional value-at-risk(CVaR) objective. We illustrate the practical relevance of our model by evaluating it on an HIV treatmentinitiation problem, where patients aim to balance the potential benefits and risks of the treatment.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Optimizing Quantiles in Preference-based Markov Decision Processes
In the Markov decision process model, policies are usually evaluated by expected cumulative rewards. As this decision criterion is not always suitable, we propose in this paper an algorithm for computing a policy optimal…
Conformal Prediction Intervals for Markov Decision Process Trajectories
Before delegating a task to an autonomous system, a human operator may want a guarantee about the behavior of the system. This paper extends previous work on conformal prediction for functional data and conformalized qua…
Conformal PredictionManagementPredictionPrediction Intervals+2Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis
In Markov decision processes (MDPs), quantile risk measures such as Value-at-Risk are a standard metric for modeling RL agents' preferences for certain outcomes. This paper proposes a new Q-learning algorithm for quantil…
Q-LearningOptimistic Reinforcement Learning with Quantile Objectives
Reinforcement Learning (RL) has achieved tremendous success in recent years. However, the classical foundations of RL do not account for the risk sensitivity of the objective function, which is critical in various fields…
Reinforcement LearningUniversal Off-Policy Evaluation
When faced with sequential decision-making problems, it is often useful to be able to predict what would happen if decisions were made using a new policy. Those predictions must often be based on data collected under som…
counterfactualDecision MakingOff-policy evaluationSequential Decision Making