Offline Reinforcement Learning with Causal Structured World Models
Model-based methods have recently shown promising for offline reinforcement learning (RL), aiming to learn good policies from historical data without interacting with the environment. Previous model-based offline RL methods learn fully connected nets as world-models that map the states and actions to the next-step states. However, it is sensible that a world-model should adhere to the underlying causal effect such that it will support learning an effective policy generalizing well in unseen states. In this paper, We first provide theoretical results that causal world-models can outperform plain world-models for offline RL by incorporating the causal structure into the generalization error bound. We then propose a practical algorithm, oFfline mOdel-based reinforcement learning with CaUsal Structure (FOCUS), to illustrate the feasibility of learning and leveraging causal structure in offline RL. Experimental results on two benchmarks show that FOCUS reconstructs the underlying causal structure accurately and robustly. Consequently, it performs better than the plain model-based offline RL algorithms and other causal model-based RL algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
Model-based Reinforcement LearningOffline RLreinforcement-learningReinforcement LearningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Driving
In the domain of autonomous driving, the offline Reinforcement Learning~(RL) approaches exhibit notable efficacy in addressing sequential decision-making problems from offline datasets. However, maintaining safety in div…
Autonomous DrivingAutonomous VehiclesDecision MakingOffline RL+4Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning?
Causal confusion is a phenomenon where an agent learns a policy that reflects imperfect spurious correlations in the data. Such a policy may falsely appear to be optimal during training if most of the training data conta…
reinforcement-learningReinforcement LearningCausal Decision Transformer for Recommender Systems via Offline Reinforcement Learning
Reinforcement learning-based recommender systems have recently gained popularity. However, the design of the reward function, on which the agent relies to optimize its recommendation policy, is often not straightforward.…
Recommendation Systemsreinforcement-learningReinforcement LearningCausal prompting model-based offline reinforcement learning
Model-based offline Reinforcement Learning (RL) allows agents to fully utilise pre-collected datasets without requiring additional or unethical explorations. However, applying model-based offline RL to online systems pre…
modelOffline RLreinforcement-learningReinforcement Learning+1Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications
Integrating causal inference (CI) with reinforcement learning (RL) has emerged as a powerful paradigm to address critical limitations in classical RL, including low explainability, lack of robustness and generalization f…
Representation LearningReinforcement LearningTransfer LearningCausal Inference