Learning Goal-Conditioned Policies from Sub-Optimal Offline Data via Metric Learning
We address the problem of learning optimal behavior from sub-optimal datasets for goal-conditioned offline reinforcement learning. To do so, we propose the use of metric learning to approximate the optimal value function for goal-conditioned offline RL problems under sparse rewards, invertible actions and deterministic transitions. We introduce distance monotonicity, a property for representations to recover optimality and propose an optimization objective that leads to such property. We use the proposed value function to guide the learning of a policy in an actor-critic fashion, a method we name MetricRL. Experimentally, we show that our method estimates optimal behaviors from severely sub-optimal offline datasets without suffering from out-of-distribution estimation errors. We demonstrate that MetricRL consistently outperforms prior state-of-the-art goal-conditioned RL methods in learning optimal policies from sub-optimal offline datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Metric LearningOffline RLreinforcement-learningSimilar Papers 제목 키워드 기반
Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL
Offline Goal-Conditioned Reinforcement Learning (Offline GCRL) is an important problem in RL that focuses on acquiring diverse goal-oriented skills solely from pre-collected behavior datasets. In this setting, the reward…
Offline RLSTO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
Offline reinforcement learning (RL) enables policy learning from pre-collected datasets, avoiding costly and risky online interactions, but it often struggles with long-horizon tasks involving sparse rewards. Existing go…
Reinforcement LearningOffline RLEfficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning
Offline goal-conditioned reinforcement learning (GCRL) is a practical reinforcement learning paradigm that aims to learn goal-conditioned policies from reward-free offline data. Despite recent advances in hierarchical ar…
Reinforcement LearningGoal-Conditioned Data Augmentation for Offline Reinforcement Learning
Offline reinforcement learning (RL) enables policy learning from pre-collected offline datasets, relaxing the need to interact directly with the environment. However, limited by the quality of offline datasets, it genera…
D4RLData AugmentationOffline RLreinforcement-learning+3Goal-conditioned Offline Reinforcement Learning through State Space Partitioning
Offline reinforcement learning (RL) aims to infer sequential decision policies using only offline datasets. This is a particularly difficult setup, especially when learning to achieve multiple different goals or outcomes…
Inductive Biasreinforcement-learningReinforcement LearningReinforcement Learning (RL)