Model-Invariant State Abstractions for Model-Based Reinforcement Learning

2021-02-19 · ICLR Workshop SSL-RL 2021 5 · Manan Tomar, Amy Zhang, Roberto Calandra, Matthew E. Taylor, Joelle Pineau

Accuracy and generalization of dynamics models is key to the success of model-based reinforcement learning (MBRL). As the complexity of tasks increases, so does the sample inefficiency of learning accurate dynamics models. However, many complex tasks also exhibit sparsity in the dynamics, i.e., actions have only a local effect on the system dynamics. In this paper, we exploit this property with a causal invariance perspective in the single-task setting, introducing a new type of state abstraction called \textit{model-invariance}. Unlike previous forms of state abstractions, a model-invariance state abstraction leverages causal sparsity over state variables. This allows for compositional generalization to unseen states, something that non-factored forms of state abstractions cannot do. We prove that an optimal policy can be learned over this model-invariance state abstraction and show improved generalization in a simple toy domain. Next, we propose a practical method to approximately learn a model-invariant representation for complex domains and validate our approach by showing improved modelling performance over standard maximum likelihood approaches on challenging tasks, such as the MuJoCo-based Humanoid. Finally, within the MBRL setting we show strong performance gains with respect to sample efficiency across a host of other continuous control tasks.

📄 PDF Abstract BibTeX arXiv:2102.09850

Code (0)

등록된 구현이 없습니다.

Tasks

continuous-control Continuous ControlmodelModel-based Reinforcement LearningMuJoCoreinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Invariant Causal Prediction for Block MDPs

2020-03-12 · ICML 2020 1 · Amy Zhang, Clare Lyle, Shagun Sodhani, Angelos Filos 외

Generalization across environments is critical to the successful application of reinforcement learning algorithms to real-world challenges. In this paper, we consider the problem of learning abstractions that generalize …

Causal Inference Prediction Reinforcement Learning Variable Selection

State Abstractions for Lifelong Reinforcement Learning

2018-07-01 · ICML 2018 7 · David Abel, Dilip Arumugam, Lucas Lehnert, Michael Littman

In lifelong reinforcement learning, agents must effectively transfer knowledge across tasks while simultaneously addressing exploration, credit assignment, and generalization. State abstraction can help overcome the…

reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

ConTraIRL: Factorized Contrastive Abstractions for Transferable IRL

2026-06-02 · Yikang Gui, Bikramjit Banerjee, Prashant Doshi arxiv

Reward transfer in Inverse Reinforcement Learning (IRL) is unreliable when policies must generalize to unseen combinations of environment dynamics and task goals. We propose Factorized Contrastive Abstractions for Transf…

Reinforcement Learning Continuous Control

Hierarchical and Permutation-Invariant Feature Transformation Learning via Policy-Guided Embedding Search

2026-09-09 · Rui Liu, Tao Zhe, Yanyong Huang, Sankha Narayan Guria 외 arxiv

Feature transformation improves predictive performance on tabular data by constructing informative abstractions from raw features. Recent generative approaches encode transformation knowledge into continuous embedding sp…

Reinforcement Learning

Learning Dynamic Abstract Representations for Sample-Efficient Reinforcement Learning

2022-10-04 · Mehdi Dadvar, Rashmeet Kaur Nayyar, Siddharth Srivastava

In many real-world problems, the learning agent needs to learn a problem's abstractions and solution simultaneously. However, most such abstractions need to be designed and refined by hand for different problems and doma…