SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigation
We propose SplitNet, a method for decoupling visual perception and policy learning. By incorporating auxiliary tasks and selective learning of portions of the model, we explicitly decompose the learning objectives for visual navigation into perceiving the world and acting on that perception. We show dramatic improvements over baseline models on transferring between simulators, an encouraging step towards Sim2Real. Additionally, SplitNet generalizes better to unseen environments from the same simulator and transfers faster and more effectively to novel embodied navigation tasks. Further, given only a small sample from a target domain, SplitNet can match the performance of traditional end-to-end pipelines which receive the entire dataset. Code is available https://github.com/facebookresearch/splitnet
Code (1)
Tasks
Visual NavigationSimilar Papers 제목 키워드 기반
SplitNet: Learnable Clean-Noisy Label Splitting for Learning with Noisy Labels
Annotating the dataset with high-quality labels is crucial for performance of deep network, but in real world scenarios, the labels are often contaminated by noise. To address this, some methods were proposed to automati…
Learning with noisy labelsSplitNets: Designing Neural Architectures for Efficient Distributed Computing on Head-Mounted Systems
We design deep neural networks (DNNs) and corresponding networks' splittings to distribute DNNs' workload to camera sensors and a centralized aggregator on head mounted devices to meet system performance targets in infer…
3D ClassificationDistributed ComputingNeural Architecture SearchEmbodied Multimodal Multitask Learning
Recent efforts on training visual navigation agents conditioned on language using deep reinforcement learning have been successful in learning policies for different multimodal tasks, such as semantic goal navigation and…
Deep Reinforcement LearningDisentanglementEmbodied Question AnsweringQuestion Answering+3PaLM-E: An Embodied Multimodal Language Model
Large language models excel at a wide range of complex tasks. However, enabling general inference in the real world, e.g., for robotics problems, raises the challenge of grounding. We propose embodied language models to …
Language ModelingLanguage ModellingLarge Language Modelmodel+4SplitNet: Learning to Semantically Split Deep Networks for Parameter Reduction and Model Parallelization
We propose a novel deep neural network that is both lightweight and effectively structured for model parallelization. Our network, which we name as SplitNet, automatically learns to split the network weights into ei…
General Classificationimage-classificationImage Classification