Stable Reinforcement Learning with Unbounded State Space
We consider the problem of reinforcement learning (RL) with unbounded state space motivated by the classical problem of scheduling in a queueing network. Traditional policies as well as error metric that are designed for finite, bounded or compact state space, require infinite samples for providing any meaningful performance guarantee (e.g. $\ell_\infty$ error) for unbounded state space. That is, we need a new notion of performance metric. As the main contribution of this work, inspired by the literature in queuing systems and control theory, we propose stability as the notion of "goodness": the state dynamics under the policy should remain in a bounded region with high probability. As a proof of concept, we propose an RL policy using Sparse-Sampling-based Monte Carlo Oracle and argue that it satisfies the stability property as long as the system dynamics under the optimal policy respects a Lyapunov function. The assumption of existence of a Lyapunov function is not restrictive as it is equivalent to the positive recurrence or stability property of any Markov chain, i.e., if there is any policy that can stabilize the system then it must possess a Lyapunov function. And, our policy does not utilize the knowledge of the specific Lyapunov function. To make our method sample efficient, we provide an improved, sample efficient Sparse-Sampling-based Monte Carlo Oracle with Lipschitz value function that may be of interest in its own right. Furthermore, we design an adaptive version of the algorithm, based on carefully constructed statistical tests, which finds the correct tuning parameter automatically.
Code (0)
등록된 구현이 없습니다.
Tasks
reinforcement-learningReinforcement LearningReinforcement Learning (RL)SchedulingSimilar Papers 제목 키워드 기반
Learning to Stabilize Online Reinforcement Learning in Unbounded State Spaces
In many reinforcement learning (RL) applications, we want policies that reach desired states and then keep the controlled system within an acceptable region around the desired states over an indefinite period of time. Th…
Attributereinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Intervention-Assisted Policy Gradient Methods for Online Stochastic Queuing Network Optimization: Technical Report
Deep Reinforcement Learning (DRL) offers a powerful approach to training neural network control policies for stochastic queuing networks (SQN). However, traditional DRL methods rely on offline simulations or static datas…
Deep Reinforcement LearningPolicy Gradient Methodsreinforcement-learningReinforcement LearningLine zonotopes: A tool for state estimation and fault diagnosis of unbounded and descriptor systems
This paper proposes new methods for set-based state estimation and active fault diagnosis (AFD) of linear descriptor systems (LDS). Unlike intervals, ellipsoids, and zonotopes, constrained zonotopes (CZs) can directly in…
Fault DiagnosisState EstimationA Memory-Based Reinforcement Learning Approach to Integrated Sensing and Communication
In this paper, we consider a point-to-point integrated sensing and communication (ISAC) system, where a transmitter conveys a message to a receiver over a channel with memory and simultaneously estimates the state of the…
Deep Reinforcement LearningIntegrated sensing and communicationISACReinforcement Learning (RL)Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithms
We introduce controlgym, a library of thirty-six industrial control settings, and ten infinite-dimensional partial differential equation (PDE)-based control problems. Integrated within the OpenAI Gym/Gymnasium (Gym) fram…
BenchmarkingOpenAI Gymreinforcement-learningReinforcement Learning+1