More for Less: Safe Policy Improvement With Stronger Performance Guarantees
In an offline reinforcement learning setting, the safe policy improvement (SPI) problem aims to improve the performance of a behavior policy according to which sample data has been generated. State-of-the-art approaches to SPI require a high number of samples to provide practical probabilistic guarantees on the improved policy's performance. We present a novel approach to the SPI problem that provides the means to require less data for such guarantees. Specifically, to prove the correctness of these guarantees, we devise implicit transformations on the data set and the underlying environment model that serve as theoretical foundations to derive tighter improvement bounds for SPI. Our empirical evaluation, using the well-established SPI with baseline bootstrapping (SPIBB) algorithm, on standard benchmarks shows that our method indeed significantly reduces the sample complexity of the SPIBB algorithm.
Code (1)
Similar Papers 제목 키워드 기반
Safe Policy Improvement with Soft Baseline Bootstrapping
Batch Reinforcement Learning (Batch RL) consists in training a policy using trajectories collected with another policy, called the behavioural policy. Safe policy improvement (SPI) provides guarantees with high probabili…
Reinforcement LearningGreedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergences
Approximate Policy Iteration (API) algorithms alternate between (approximate) policy evaluation and (approximate) greedification. Many different approaches have been explored for approximate policy evaluation, but less i…
Policy Gradient MethodsSOE: Sample-Efficient Robot Policy Self-Improvement via On-Manifold Exploration
Intelligent agents progress by continually refining their capabilities through actively exploring environments. Yet robot policies often lack sufficient exploration capability due to action mode collapse. Existing method…
Safe Reinforcement Learning for Legged Locomotion
Designing control policies for legged locomotion is complex due to the under-actuated and non-continuous robot dynamics. Model-free reinforcement learning provides promising tools to tackle this challenge. However, a maj…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement LearningSPAARS: Safer RL Policy Alignment through Abstract Exploration and Refined Exploitation of Action Space
Offline-to-online reinforcement learning (RL) offers a promising paradigm for robotics by pre-training policies on safe, offline demonstrations and fine-tuning them via online interaction. However, a fundamental challeng…
Reinforcement Learning