Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentation
Satisfying safety constraints almost surely (or with probability one) can be critical for the deployment of Reinforcement Learning (RL) in real-life applications. For example, plane landing and take-off should ideally occur with probability one. We address the problem by introducing Safety Augmented (Saute) Markov Decision Processes (MDPs), where the safety constraints are eliminated by augmenting them into the state-space and reshaping the objective. We show that Saute MDP satisfies the Bellman equation and moves us closer to solving Safe RL with constraints satisfied almost surely. We argue that Saute MDP allows viewing the Safe RL problem from a different perspective enabling new features. For instance, our approach has a plug-and-play nature, i.e., any RL algorithm can be "Sauteed". Additionally, state augmentation allows for policy generalization across safety constraints. We finally show that Saute RL algorithms can outperform their state-of-the-art counterparts when constraint satisfaction is of high importance.
Code (1)
Tasks
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement LearningSimilar Papers 제목 키워드 기반
Almost Surely $\sqrt{T}$ Regret for Adaptive LQR
The Linear-Quadratic Regulation (LQR) problem with unknown system parameters has been widely studied, but it has remained unclear whether $\tilde{ \mathcal{O}}(\sqrt{T})$ regret, which is the best known dependence on tim…
Enforcing Almost-Sure Reachability in POMDPs
Partially-Observable Markov Decision Processes (POMDPs) are a well-known stochastic model for sequential decision making under limited information. We consider the EXPTIME-hard problem of synthesising policies that almos…
Decision Makingreinforcement-learningReinforcement Learning (RL)Safe Exploration+1Assured RL: Reinforcement Learning with Almost Sure Constraints
We consider the problem of finding optimal policies for a Markov Decision Process with almost sure constraints on state transitions and action triplets. We define value and action-value functions that satisfy a barrier-b…
Q-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Constrained Cross-Entropy Method for Safe Reinforcement Learning
We study a safe reinforcement learning problem in which the constraints are defined as the expected cost over finite-length trajectories. We propose a constrained cross-entropy-based method to solve this problem. The met…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement LearningSafe Reinforcement Learning in Tensor Reproducing Kernel Hilbert Space
This paper delves into the problem of safe reinforcement learning (RL) in a partially observable environment with the aim of achieving safe-reachability objectives. In traditional partially observable Markov decision pro…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Safe Reinforcement Learning