Constraint Satisfaction Propagation: Non-stationary Policy Synthesis for Temporal Logic Planning
Problems arise when using reward functions to capture dependencies between sequential time-constrained goal states because the state-space must be prohibitively expanded to accommodate a history of successfully achieved sub-goals. Also, policies and value functions derived with stationarity assumptions are not readily decomposable, leading to a tension between reward maximization and task generalization. We demonstrate a logic-compatible approach using model-based knowledge of environment dynamics and deadline information to directly infer non-stationary policies composed of reusable stationary policies. The policies are constructed to maximize the probability of satisfying time-sensitive goals while respecting time-varying obstacles. Our approach explicitly maintains two different spaces, a high-level logical task specification where the task-variables are grounded onto the low-level state-space of a Markov decision process. Computing satisfiability at the task-level is made possible by a Bellman-like equation which operates on a tensor that links the temporal relationship between the two spaces; the equation solves for a value function that can be explicitly interpreted as the probability of sub-goal satisfaction under the synthesized non-stationary policy, an approach we term Constraint Satisfaction Propagation (CSP).
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation
We consider the offline constrained reinforcement learning (RL) problem, in which the agent aims to compute a policy that maximizes expected return while satisfying given cost constraints, learning only from a pre-collec…
Offline RLOff-policy evaluationreinforcement-learningReinforcement Learning+1Robust Model Predictive Control of Time-Delay Systems through System Level Synthesis
We present a robust model predictive control method (MPC) for discrete-time linear time-delayed systems with state and control input constraints. The system is subject to both polytopic model uncertainty and additive dis…
Model Predictive ControlFrom Cumulative Constraints to Adaptive Runtime Safety Control for Nonstationary Reinforcement Learning
Safety in reinforcement learning is often specified through cumulative cost constraints, but these trajectory-level guarantees do not directly prevent unsafe individual decisions, especially under nonstationarity. In con…
Reinforcement LearningExterior Penalty Policy Optimization with Penalty Metric Network under Constraints
In Constrained Reinforcement Learning (CRL), agents explore the environment to learn the optimal policy while satisfying constraints. The penalty function method has recently been studied as an effective approach for han…
Safe ExplorationThe Regularization of Small Sub-Constraint Satisfaction Problems
This paper describes a new approach on optimization of constraint satisfaction problems (CSPs) by means of substituting sub-CSPs with locally consistent regular membership constraints. The purpose of this approach is to …