Probabilistic Counterexample Guidance for Safer Reinforcement Learning (Extended Version)

2023-07-10 · Xiaotong Ji, Antonio Filieri

Safe exploration aims at addressing the limitations of Reinforcement Learning (RL) in safety-critical scenarios, where failures during trial-and-error learning may incur high costs. Several methods exist to incorporate external knowledge or to use proximal sensor data to limit the exploration of unsafe states. However, reducing exploration risks in unknown environments, where an agent must discover safety threats during exploration, remains challenging. In this paper, we target the problem of safe exploration by guiding the training with counterexamples of the safety requirement. Our method abstracts both continuous and discrete state-space systems into compact abstract models representing the safety-relevant knowledge acquired by the agent during exploration. We then exploit probabilistic counterexample generation to construct minimal simulation submodels eliciting safety requirement violations, where the agent can efficiently train offline to refine its policy towards minimising the risk of safety violations during the subsequent online exploration. We demonstrate our method's effectiveness in reducing safety violations during online exploration in preliminary experiments by an average of 40.3% compared with QL and DQN standard algorithms and 29.1% compared with previous related work, while achieving comparable cumulative rewards with respect to unrestricted exploration and alternative approaches.

📄 PDF Abstract BibTeX arXiv:2307.04927

Code (1)

xtji/cex-guided-rl 공식 구현 pytorch

Tasks

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Exploration

Methods 이 논문이 사용한 방법론

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Convolution A convolution is a type of matrix operation, consisting of a kernel, a small matrix of weights, that slides over input data performing element-wise multiplication with the…

Q-Learning Q-Learning is an off-policy temporal difference control algorithm: $$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} +…

DQN A DQN, or Deep Q-Network, approximates a state-value function in a Q-Learning framework with a neural network. In the Atari…

Counterexample-Driven Synthesis for Probabilistic Program Sketches

2019-04-28 · Milan Češka, Christian Hensel, Sebastian Junges, Joost-Pieter Katoen

Probabilistic programs are key to deal with uncertainty in e.g. controller synthesis. They are typically small but intricate. Their development is complex and error prone requiring quantitative reasoning over a myriad of…

When is an Example a Counterexample?

2013-10-23 · Eric Pacuit, Arthur Paul Pedersen, Jan-Willem Romeijn

In this extended abstract, we carefully examine a purported counterexample to a postulate of iterated belief revision. We suggest that the example is better seen as a failure to apply the theory of belief revision in suf…

Safe Reinforcement Learning via Probabilistic Logic Shields

2023-03-06 · Wen-Chi Yang, Giuseppe Marra, Gavin Rens, Luc De Raedt

Safe Reinforcement learning (Safe RL) aims at learning optimal policies while staying safe. A popular solution to Safe RL is shielding, which uses a logical safety specification to prevent an RL agent from taking unsafe …

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Reinforcement Learning

The Counterexample Game: Iterated Conceptual Analysis and Repair in Language Models

2026-05-05 · Daniel Drucker, Kyle Mahowald arxiv

Conceptual analysis -- proposing definitions and refining them through counterexamples -- is central to philosophical methodology. We study whether language models can perform this task through iterated analysis and repa…

Counterexample-Guided Repair of Reinforcement Learning Systems Using Safety Critics

2024-05-24 · David Boetius, Stefan Leue

Naively trained Deep Reinforcement Learning agents may fail to satisfy vital safety constraints. To avoid costly retraining, we may desire to repair a previously trained reinforcement learning agent to obviate unsafe beh…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning