Percentile Criterion Optimization in Offline Reinforcement Learning
In reinforcement learning, robust policies for high-stakes decision-making problems with limited data are usually computed by optimizing the \emph{percentile criterion}. The percentile criterion is approximately solved by constructing an \emph{ambiguity set} that contains the true model with high probability and optimizing the policy for the worst model in the set. Since the percentile criterion is non-convex, constructing ambiguity sets is often challenging. Existing work uses \emph{Bayesian credible regions} as ambiguity sets, but they are often unnecessarily large and result in learning overly conservative policies. To overcome these shortcomings, we propose a novel Value-at-Risk based dynamic programming algorithm to optimize the percentile criterion without explicitly constructing any ambiguity sets. Our theoretical and empirical results show that our algorithm implicitly constructs much smaller ambiguity sets and learns less conservative robust policies.
Code (1)
Tasks
Decision Makingreinforcement-learningReinforcement LearningSimilar Papers 제목 키워드 기반
Soft-Robust Algorithms for Batch Reinforcement Learning
In reinforcement learning, robust policies for high-stakes decision-making problems with limited data are usually computed by optimizing the percentile criterion, which minimizes the probability of a catastrophic failure…
Decision Makingreinforcement-learningReinforcement LearningReinforcement Learning (RL)Offline Reinforcement Learning for Wireless Network Optimization with Mixture Datasets
The recent development of reinforcement learning (RL) has boosted the adoption of online RL for wireless radio resource management (RRM). However, online RL algorithms require direct interactions with the environment, wh…
ManagementOffline RLQ-Learningreinforcement-learning+3Optimizing Percentile Criterion Using Robust MDPs
We address the problem of computing reliable policies in reinforcement learning problems with limited data. In particular, we compute policies that achieve good returns with high confidence when deployed. This objective,…
Reinforcement LearningReinforcement Learning (RL)Reinforcement Learning for Code Optimization
RL for code correctness is now established: have the model generate a program, run it against hidden test cases, and reward solutions that pass. Extending this to code optimization seems straightforward: just add executi…
Reinforcement LearningDataset Distillation for Offline Reinforcement Learning
Offline reinforcement learning often requires a quality dataset that we can train a policy on. However, in many situations, it is not possible to get such a dataset, nor is it easy to train a policy to perform well in th…
Dataset Distillationreinforcement-learningReinforcement Learning