Graph Backup: Data Efficient Backup Exploiting Markovian Transitions

2022-05-31 · Zhengyao Jiang, Tianjun Zhang, Robert Kirk, Tim Rocktäschel, Edward Grefenstette

The successes of deep Reinforcement Learning (RL) are limited to settings where we have a large stream of online experiences, but applying RL in the data-efficient setting with limited access to online interactions is still challenging. A key to data-efficient RL is good value estimation, but current methods in this space fail to fully utilise the structure of the trajectory data gathered from the environment. In this paper, we treat the transition data of the MDP as a graph, and define a novel backup operator, Graph Backup, which exploits this graph structure for better value estimation. Compared to multi-step backup methods such as $n$-step $Q$-Learning and TD($\lambda$), Graph Backup can perform counterfactual credit assignment and gives stable value estimates for a state regardless of which trajectory the state is sampled from. Our method, when combined with popular value-based methods, provides improved performance over one-step and multi-step methods on a suite of data-efficient RL benchmarks including MiniGrid, Minatar and Atari100K. We further analyse the reasons for this performance boost through a novel visualisation of the transition graphs of Atari games.

📄 PDF Abstract BibTeX arXiv:2205.15824

Code (1)

ZhengyaoJiang/graphbackup 공식 구현 pytorch

Tasks

Atari GamescounterfactualDeep Reinforcement LearningQ-LearningReinforcement Learning (RL)

Backup Control Barrier Functions: Formulation and Comparative Study

2021-04-22 · Yuxiao Chen, Mrdjan Jankovic, Mario Santillo, Aaron D. Ames

The backup control barrier function (CBF) was recently proposed as a tractable formulation that guarantees the feasibility of the CBF quadratic programming (QP) via an implicitly defined control invariant set. The contro…

Math

Backup-Based Safety Filters: A Comparative Review of Backup CBF, Model Predictive Shielding, and gatekeeper

2026-04-02 · Taekyung Kim, Aswin D. Menon, Akshunn Trivedi, Dimitra Panagou arxiv

This paper revisits three backup-based safety filters -- Backup Control Barrier Functions (Backup CBF), Model Predictive Shielding (MPS), and gatekeeper -- through a unified comparative framework. Using a common safety-f…

Convergent Tree Backup and Retrace with Function Approximation

2017-05-25 · ICML 2018 7 · Ahmed Touati, Pierre-Luc Bacon, Doina Precup, Pascal Vincent

Off-policy learning is key to scaling up reinforcement learning as it allows to learn about a target policy from the experience generated by a different behavior policy. Unfortunately, it has been challenging to combine …

Reinforcement Learning

Safe Exploration in Reinforcement Learning: Training Backup Control Barrier Functions with Zero Training Time Safety Violations

2023-12-13 · Pedram Rabiee, Amirsaeid Safari

This paper introduces the reinforcement learning backup shield (RLBUS), an algorithm that guarantees safe exploration in reinforcement learning (RL) by incorporating backup control barrier functions (BCBFs). RLBUS constr…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)Safe Exploration+1

Deep Reinforcement Learning for Backup Strategies against Adversaries

2021-02-12 · Pascal Debus, Nicolas Müller, Konstantin Böttinger

Many defensive measures in cyber security are still dominated by heuristics, catalogs of standard procedures, and best practices. Considering the case of data backup strategies, we aim towards mathematically modeling the…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)