Visual Reinforcement Learning with Self-Supervised 3D Representations
A prominent approach to visual Reinforcement Learning (RL) is to learn an internal state representation using self-supervised methods, which has the potential benefit of improved sample-efficiency and generalization through additional learning signal and inductive biases. However, while the real world is inherently 3D, prior efforts have largely been focused on leveraging 2D computer vision techniques as auxiliary self-supervision. In this work, we present a unified framework for self-supervised learning of 3D representations for motor control. Our proposed framework consists of two phases: a pretraining phase where a deep voxel-based 3D autoencoder is pretrained on a large object-centric dataset, and a finetuning phase where the representation is jointly finetuned together with RL on in-domain data. We empirically show that our method enjoys improved sample efficiency in simulated manipulation tasks compared to 2D representation learning methods. Additionally, our learned policies transfer zero-shot to a real robot setup with only approximate geometric correspondence, and successfully solve motor control tasks that involve grasping and lifting from a single, uncalibrated RGB camera. Code and videos are available at https://yanjieze.com/3d4rl/ .
Code (1)
Tasks
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Representation LearningSelf-Supervised LearningSimilar Papers 제목 키워드 기반
Scaling and Benchmarking Self-Supervised Visual Representation Learning
Self-supervised learning aims to learn representations from the data itself without explicit manual supervision. Existing efforts ignore a crucial aspect of self-supervised learning - the ability to scale to large amount…
Benchmarkingobject-detectionObject DetectionReinforcement Learning+4Masked Visual Pre-training for Motor Control
This paper shows that self-supervised visual pre-training from real-world images is effective for learning motor control tasks from pixels. We first train the visual representations by masked modeling of natural images. …
Robot Manipulation GeneralizationState EstimationSelf-supervised Hierarchical Visual Reasoning with World Model
3D open-world environments with adversarial opponents remain a core challenge for reinforcement learning due to their vast state spaces. Effective reasoning representations are essential in such settings. While existing …
Reinforcement LearningVisual ReasoningLocal-Guided Global: Paired Similarity Representation for Visual Reinforcement Learning
Recent vision-based reinforcement learning (RL) methods have found extracting high-level features from raw pixels with self-supervised learning to be effective in learning policies. However, these methods focus on le…
Atari Gamesreinforcement-learningReinforcement LearningReinforcement Learning (RL)+2Light-weight probing of unsupervised representations for Reinforcement Learning
Unsupervised visual representation learning offers the opportunity to leverage large corpora of unlabeled trajectories to form useful visual representations, which can benefit the training of reinforcement learning (RL) …
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Representation Learning+1