Constrained Exploration in Reinforcement Learning with Optimality Preservation

2023-04-05 · Peter C. Y. Chen

We consider a class of reinforcement-learning systems in which the agent follows a behavior policy to explore a discrete state-action space to find an optimal policy while adhering to some restriction on its behavior. Such restriction may prevent the agent from visiting some state-action pairs, possibly leading to the agent finding only a sub-optimal policy. To address this problem we introduce the concept of constrained exploration with optimality preservation, whereby the exploration behavior of the agent is constrained to meet a specification while the optimality of the (original) unconstrained learning process is preserved. We first establish a feedback-control structure that models the dynamics of the unconstrained learning process. We then extend this structure by adding a supervisor to ensure that the behavior of the agent meets the specification, and establish (for a class of reinforcement-learning problems with a known deterministic environment) a necessary and sufficient condition under which optimality is preserved. This work demonstrates the utility and the prospect of studying reinforcement-learning problems in the context of the theories of discrete-event systems, automata and formal languages.

📄 PDF Abstract BibTeX arXiv:2304.03104

Code (0)

등록된 구현이 없습니다.

Tasks

reinforcement-learning Reinforcement Learning

Adaptive trajectory-constrained exploration strategy for deep reinforcement learning

2023-12-27 · GuoJian Wang, Faguo Wu, Xiao Zhang, Ning Guo 외

Deep reinforcement learning (DRL) faces significant challenges in addressing the hard-exploration problems in tasks with sparse or deceptive rewards and large state spaces. These challenges severely limit the practical a…

Deep Reinforcement Learning MuJoCo Multi-agent Reinforcement Learning reinforcement-learning+1

Safe Exploration Using Bayesian World Models and Log-Barrier Optimization

2024-05-09 · Yarden As, Bhavya Sukhija, Andreas Krause

A major challenge in deploying reinforcement learning in online tasks is ensuring that safety is maintained throughout the learning process. In this work, we propose CERL, a new method for solving constrained Markov deci…

Safe Exploration

Safe Exploration in Reinforcement Learning: A Generalized Formulation and Algorithms

2023-10-05 · NeurIPS 2023 11

Safe exploration is essential for the practical use of reinforcement learning (RL) in many real-world scenarios. In this paper, we present a generalized safe exploration (GSE) problem as a unified formulation of common s…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Exploration

Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance

2019-11-16 · Mingxuan Jing, Xiaojian Ma, Wenbing Huang, Fuchun Sun 외

In this paper, we study Reinforcement Learning from Demonstrations (RLfD) that improves the exploration efficiency of Reinforcement Learning (RL) by providing expert demonstrations. Most of existing RLfD methods require …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Efficient Exploration Using Extra Safety Budget in Constrained Policy Optimization

2023-02-28 · Haotian Xu, Shengjie Wang, Zhaolei Wang, Yunzhe Zhang 외

Reinforcement learning (RL) has achieved promising results on most robotic control tasks. Safety of learning-based controllers is an essential notion of ensuring the effectiveness of the controllers. Current methods adop…

Efficient Exploration Reinforcement Learning (RL)