BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learning
Offline model-based reinforcement learning (MBRL) enhances data efficiency by utilizing pre-collected datasets to learn models and policies, especially in scenarios where exploration is costly or infeasible. Nevertheless, its performance often suffers from the objective mismatch between model and policy learning, resulting in inferior performance despite accurate model predictions. This paper first identifies the primary source of this mismatch comes from the underlying confounders present in offline data for MBRL. Subsequently, we introduce \textbf{B}ilin\textbf{E}ar \textbf{CAUS}al r\textbf{E}presentation~(BECAUSE), an algorithm to capture causal representation for both states and actions to reduce the influence of the distribution shift, thus mitigating the objective mismatch problem. Comprehensive evaluations on 18 tasks that vary in data quality and environment context demonstrate the superior performance of BECAUSE over existing offline RL algorithms. We show the generalizability and robustness of BECAUSE under fewer samples or larger numbers of confounders. Additionally, we offer theoretical analysis of BECAUSE to prove its error bound and sample efficiency when integrating causal representation into offline MBRL.
Code (0)
등록된 구현이 없습니다.
Tasks
Model-based Reinforcement LearningOffline RLSimilar Papers 제목 키워드 기반
Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Driving
In the domain of autonomous driving, the offline Reinforcement Learning~(RL) approaches exhibit notable efficacy in addressing sequential decision-making problems from offline datasets. However, maintaining safety in div…
Autonomous DrivingAutonomous VehiclesDecision MakingOffline RL+4Generalizable Information Theoretic Causal Representation
It is evidence that representation learning can improve model's performance over multiple downstream tasks in many real-world scenarios, such as image classification and recommender systems. Existing learning approaches …
counterfactualimage-classificationImage ClassificationRecommendation Systems+1Informative Robust Causal Representation for Generalizable Deep Learning
In many real-world scenarios, such as image classification and recommender systems, it is evidence that representation learning can improve model's performance over multiple downstream tasks. Existing learning approaches…
counterfactualDeep Learningimage-classificationImage Classification+2Unifying Causal Reinforcement Learning: Survey, Taxonomy, Algorithms and Applications
Integrating causal inference (CI) with reinforcement learning (RL) has emerged as a powerful paradigm to address critical limitations in classical RL, including low explainability, lack of robustness and generalization f…
Representation LearningReinforcement LearningTransfer LearningCausal InferenceBilinear Convolution Decomposition for Causal RL Interpretability
Efforts to interpret reinforcement learning (RL) models often rely on high-level techniques such as attribution or probing, which provide only correlational insights and coarse causal control. This work proposes replacin…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)