Assured RL: Reinforcement Learning with Almost Sure Constraints

2020-12-24 · Agustin Castellano, Juan Bazerque, Enrique Mallada

We consider the problem of finding optimal policies for a Markov Decision Process with almost sure constraints on state transitions and action triplets. We define value and action-value functions that satisfy a barrier-based decomposition which allows for the identification of feasible policies independently of the reward process. We prove that, given a policy {\pi}, certifying whether certain state-action pairs lead to feasible trajectories under {\pi} is equivalent to solving an auxiliary problem aimed at finding the probability of performing an unfeasible transition. Using this interpretation,we develop a Barrier-learning algorithm, based on Q-Learning, that identifies such unsafe state-action pairs. Our analysis motivates the need to enhance the Reinforcement Learning (RL) framework with an additional signal, besides rewards, called here damage function that provides feasibility information and enables the solution of RL problems with model-free constraints. Moreover, our Barrier-learning algorithm wraps around existing RL algorithms, such as Q-Learning and SARSA, giving them the ability to solve almost-surely constrained problems.

📄 PDF Abstract BibTeX arXiv:2012.13036

Code (0)

등록된 구현이 없습니다.

Tasks

Q-Learningreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Methods 이 논문이 사용한 방법론

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

Assured Learning-enabled Autonomy: A Metacognitive Reinforcement Learning Framework

2021-03-23 · Aquib Mustafa, Majid Mazouchi, Subramanya Nageshrao, Hamidreza Modares

Reinforcement learning (RL) agents with pre-specified reward functions cannot provide guaranteed safety across variety of circumstances that an uncertain system might encounter. To guarantee performance while assuring sa…

Decision Making reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentation

2022-02-14 · Aivar Sootla, Alexander I. Cowen-Rivers, Taher Jafferjee, Ziyan Wang 외

Satisfying safety constraints almost surely (or with probability one) can be critical for the deployment of Reinforcement Learning (RL) in real-life applications. For example, plane landing and take-off should ideally oc…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

Run Time Assured Reinforcement Learning for Six Degree-of-Freedom Spacecraft Inspection

2024-06-17 · Kyle Dunlap, Kochise Bennett, David van Wijk, Nathaniel Hamilton 외

The trial and error approach of reinforcement learning (RL) results in high performance across many complex tasks, but it can also lead to unsafe behavior. Run time assurance (RTA) approaches can be used to assure safety…

Reinforcement Learning (RL)

FAST: A Fairness Assured Service Recommendation Strategy Considering Service Capacity Constraint

2020-12-02 · Yao Wu, Jian Cao, Guandong Xu

An excessive number of customers often leads to a degradation in service quality. However, the capacity constraints of services are ignored by recommender systems, which may lead to unsatisfactory recommendation. This pr…

Fairness Recommendation Systems

Systems Theoretic Process Analysis of a Run Time Assured Neural Network Control System

2022-09-01 · Kerianne L. Hobbs, Benjamin K. Heiner, Lillian Busse, Kyle Dunlap 외

This research considers the problem of identifying safety constraints and developing Run Time Assurance (RTA) for Deep Reinforcement Learning (RL) Tactical Autopilots that use neural network control systems (NNCS). This …

Collision Avoidance Deep Reinforcement Learning Reinforcement Learning (RL)