Contrastive Value Learning: Implicit Models for Simple Offline RL
Model-based reinforcement learning (RL) methods are appealing in the offline setting because they allow an agent to reason about the consequences of actions without interacting with the environment. Prior methods learn a 1-step dynamics model, which predicts the next state given the current state and action. These models do not immediately tell the agent which actions to take, but must be integrated into a larger RL framework. Can we model the environment dynamics in a different way, such that the learned model does directly indicate the value of each action? In this paper, we propose Contrastive Value Learning (CVL), which learns an implicit, multi-step model of the environment dynamics. This model can be learned without access to reward functions, but nonetheless can be used to directly estimate the value of each action, without requiring any TD learning. Because this model represents the multi-step transitions implicitly, it avoids having to predict high-dimensional observations and thus scales to high-dimensional tasks. Our experiments demonstrate that CVL outperforms prior offline RL methods on complex continuous control benchmarks.
Code (0)
등록된 구현이 없습니다.
Tasks
continuous-controlContinuous ControlModel-based Reinforcement LearningOffline RLReinforcement Learning (RL)Similar Papers 제목 키워드 기반
VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training
Reward and representation learning are two long-standing challenges for learning an expanding set of robot manipulation skills from sensory observations. Given the inherent cost and scarcity of in-domain, task-specific r…
Offline RLOpen-Ended Question AnsweringRepresentation LearningRobot ManipulationContrastive State Augmentations for Reinforcement Learning-Based Recommender Systems
Learning reinforcement learning (RL)-based recommenders from historical user-item interaction sequences is vital to generate high-reward recommendations and improve long-term cumulative benefits. However, existing RL rec…
Recommendation Systemsreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning
Offline Reinforcement Learning (RL) has emerged as a powerful alternative to imitation learning for behavior modeling in various domains, particularly in complex navigation tasks. An existing challenge with Offline RL is…
Imitation LearningOffline RLQ-LearningQuantization+1Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
Offline reinforcement learning (RL) has received considerable attention in recent years due to its attractive capability of learning policies from offline datasets without environmental interactions. Despite some success…
ManagementMuJoCoMulti-agent Reinforcement LearningReinforcement Learning (RL)+2Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Words
The performance of sentence encoders can be significantly improved through the simple practice of fine-tuning using contrastive loss. A natural question arises: what characteristics do models acquire during contrastive l…
Contrastive LearningSentence