Causal Dynamics Learning for Task-Independent State Abstraction
Learning dynamics models accurately is an important goal for Model-Based Reinforcement Learning (MBRL), but most MBRL methods learn a dense dynamics model which is vulnerable to spurious correlations and therefore generalizes poorly to unseen states. In this paper, we introduce Causal Dynamics Learning for Task-Independent State Abstraction (CDL), which first learns a theoretically proved causal dynamics model that removes unnecessary dependencies between state variables and the action, thus generalizing well to unseen states. A state abstraction can then be derived from the learned dynamics, which not only improves sample efficiency but also applies to a wider range of tasks than existing state abstraction methods. Evaluated on two simulated environments and downstream tasks, both the dynamics model and policies learned by the proposed method generalize well to unseen states and the derived state abstraction improves sample efficiency compared to learning without it.
Code (1)
Tasks
Model-based Reinforcement LearningSimilar Papers 제목 키워드 기반
Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning
Two desiderata of reinforcement learning (RL) algorithms are the ability to learn from relatively little experience and the ability to learn policies that generalize to a range of problem specifications. In factored stat…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Model-Invariant State Abstractions for Model-Based Reinforcement Learning
Accuracy and generalization of dynamics models is key to the success of model-based reinforcement learning (MBRL). As the complexity of tasks increases, so does the sample inefficiency of learning accurate dynamics model…
continuous-controlContinuous ControlmodelModel-based Reinforcement Learning+4Invariant Causal Prediction for Block MDPs
Generalization across environments is critical to the successful application of reinforcement learning algorithms to real-world challenges. In this paper, we consider the problem of learning abstractions that generalize …
Causal InferencePredictionReinforcement LearningVariable SelectionCausal Optimal Transport of Abstractions
Causal abstraction (CA) theory establishes formal criteria for relating multiple structural causal models (SCMs) at different levels of granularity by defining maps between them. These maps have significant relevance for…
Data AugmentationTowards Identifiability of Hierarchical Temporal Causal Representation Learning
Modeling hierarchical latent dynamics behind time series data is critical for capturing temporal dependencies across multiple levels of abstraction in real-world tasks. However, existing temporal causal representation le…
Representation Learning