Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels
Controlling artificial agents from visual sensory data is an arduous task. Reinforcement learning (RL) algorithms can succeed but require large amounts of interactions between the agent and the environment. To alleviate the issue, unsupervised RL proposes to employ self-supervised interaction and learning, for adapting faster to future tasks. Yet, as shown in the Unsupervised RL Benchmark (URLB; Laskin et al. 2021), whether current unsupervised strategies can improve generalization capabilities is still unclear, especially in visual control settings. In this work, we study the URLB and propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent, and a task-aware fine-tuning strategy combined with a new proposed hybrid planner, Dyna-MPC, to adapt the agent for downstream tasks. On URLB, our method obtains 93.59% overall normalized performance, surpassing previous baselines by a staggering margin. The approach is empirically evaluated through a large-scale empirical study, which we use to validate our design choices and analyze our models. We also show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation. Project website: https://masteringurlb.github.io/
Code (1)
Tasks
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Unsupervised Reinforcement LearningSimilar Papers 제목 키워드 기반
CURL: Contrastive Unsupervised Representation Learning for Reinforcement Learning
Reinforcement Learning for control tasks where the agent learns from raw high dimensional pixels has proven to be difficult and sample-inefficient. Operating on high-dimensional observational input poses a challenging c…
Contrastive Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)+1Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning
We present DrQ-v2, a model-free reinforcement learning (RL) algorithm for visual continuous control. DrQ-v2 builds on DrQ, an off-policy actor-critic approach that uses data augmentation to learn directly from pixels. We…
continuous-controlContinuous ControlData AugmentationGPU+4CURL: Contrastive Unsupervised Representations for Reinforcement Learning
We present CURL: Contrastive Unsupervised Representations for Reinforcement Learning. CURL extracts high-level features from raw pixels using contrastive learning and performs off-policy control on top of the extracted f…
Atari GamesAtari Games 100kContinuous ControlContrastive Learning+3Bellman Meets Lyapunov: Unsupervised Reinforcement Learning via Mastering Chaos
Reinforcement learning (RL) is a powerful paradigm for training agents, yet its success rests on domain expertise of human engineers who design informative reward signals for every new task. Unsupervised RL aims to reduc…
Reinforcement LearningMastering Diverse Domains through World Models
Developing a general algorithm that learns to solve tasks across a wide range of applications has been a fundamental challenge in artificial intelligence. Although current reinforcement learning algorithms can be readily…
Atari Games 100kDecision MakingMinecraftreinforcement-learning+2