Revisiting Bisimulation Metric for Robust Representations in Reinforcement Learning
Bisimulation metric has long been regarded as an effective control-related representation learning technique in various reinforcement learning tasks. However, in this paper, we identify two main issues with the conventional bisimulation metric: 1) an inability to represent certain distinctive scenarios, and 2) a reliance on predefined weights for differences in rewards and subsequent states during recursive updates. We find that the first issue arises from an imprecise definition of the reward gap, whereas the second issue stems from overlooking the varying importance of reward difference and next-state distinctions across different training stages and task settings. To address these issues, by introducing a measure for state-action pairs, we propose a revised bisimulation metric that features a more precise definition of reward gap and novel update operators with adaptive coefficient. We also offer theoretical guarantees of convergence for our proposed metric and its improved representation distinctiveness. In addition to our rigorous theoretical analysis, we conduct extensive experiments on two representative benchmarks, DeepMind Control and Meta-World, demonstrating the effectiveness of our approach.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningReinforcement LearningSimilar Papers 제목 키워드 기반
Efficient Embedding of Semantic Similarity in Control Policies via Entangled Bisimulation
Learning generalizeable policies from visual input in the presence of visual distractions is a challenging problem in reinforcement learning. Recently, there has been renewed interest in bisimulation metrics as a tool to…
Data AugmentationReinforcement Learning (RL)Semantic SimilaritySemantic Textual SimilarityRobust Representation Learning by Clustering with Bisimulation Metrics for Visual Reinforcement Learning with Distractions
Recent work has shown that representation learning plays a critical role in sample-efficient reinforcement learning (RL) from pixels. Unfortunately, in real-world scenarios, representation learning is usually fragile to …
ClusteringReinforcement Learning (RL)Representation LearningInvariant Representations for Reinforcement Learning without Reconstruction
We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction. Our goal is to learn representations …
Causal InferenceMuJoCoreinforcement-learningReinforcement Learning+2Learning Invariant Representations for Reinforcement Learning without Reconstruction
We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction. Our goal is to learn representations …
Causal InferenceMuJoCoreinforcement-learningReinforcement Learning+2Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learning
While bisimulation-based approaches hold promise for learning robust state representations for Reinforcement Learning (RL) tasks, their efficacy in offline RL tasks has not been up to par. In some instances, their perfo…