Safe Policy Improvement with an Estimated Baseline Policy
Previous work has shown the unreliability of existing algorithms in the batch Reinforcement Learning setting, and proposed the theoretically-grounded Safe Policy Improvement with Baseline Bootstrapping (SPIBB) fix: reproduce the baseline policy in the uncertain state-action pairs, in order to control the variance on the trained policy performance. However, in many real-world applications such as dialogue systems, pharmaceutical tests or crop management, data is collected under human supervision and the baseline remains unknown. In this paper, we apply SPIBB algorithms with a baseline estimate built from the data. We formally show safe policy improvement guarantees over the true baseline even without direct access to it. Our empirical experiments on finite and continuous states tasks support the theoretical findings. It shows little loss of performance in comparison with SPIBB when the baseline policy is given, and more importantly, drastically and significantly outperforms competing algorithms both in safe policy improvement, and in average performance.
Code (0)
등록된 구현이 없습니다.
Tasks
ManagementReinforcement LearningSimilar Papers 제목 키워드 기반
CSPI-MT: Calibrated Safe Policy Improvement with Multiple Testing for Threshold Policies
When modifying existing policies in high-risk settings, it is often necessary to ensure with high certainty that the newly proposed policy improves upon a baseline, such as the status quo. In this work, we consider the p…
Safe Policy Improvement with Soft Baseline Bootstrapping
Batch Reinforcement Learning (Batch RL) consists in training a policy using trajectories collected with another policy, called the behavioural policy. Safe policy improvement (SPI) provides guarantees with high probabili…
Reinforcement LearningMulti-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs
We study the problem of Safe Policy Improvement (SPI) under constraints in the offline Reinforcement Learning (RL) setting. We consider the scenario where: (i) we have a dataset collected under a known baseline policy, (…
Reinforcement Learning (RL)Confounding-Robust Policy Improvement
We study the problem of learning personalized decision policies from observational data while accounting for possible unobserved confounding. Previous approaches, which assume unconfoundedness, i.e., that no unobserved c…
Causal InferenceSafe Policy Improvement by Minimizing Robust Baseline Regret
An important problem in sequential decision-making under uncertainty is to use limited data to compute a safe policy, i.e., a policy that is guaranteed to perform at least as well as a given baseline strategy. In this pa…
Decision MakingDecision Making Under UncertaintySequential Decision Making