Learning to Solve Voxel Building Embodied Tasks from Pixels and Natural Language Instructions
The adoption of pre-trained language models to generate action plans for embodied agents is a promising research strategy. However, execution of instructions in real or simulated environments requires verification of the feasibility of actions as well as their relevance to the completion of a goal. We propose a new method that combines a language model and reinforcement learning for the task of building objects in a Minecraft-like environment according to the natural language instructions. Our method first generates a set of consistently achievable sub-goals from the instructions and then completes associated sub-tasks with a pre-trained RL policy. The proposed method formed the RL baseline at the IGLU 2022 competition.
Code (1)
Tasks
Language ModelingLanguage ModellingMinecraftreinforcement-learningReinforcement Learning (RL)Similar Papers 제목 키워드 기반
Differentiable Registration of Images and LiDAR Point Clouds with VoxelPoint-to-Pixel Matching
Cross-modality registration between 2D images from cameras and 3D point clouds from LiDARs is a crucial task in computer vision and robotic. Previous methods estimate 2D-3D correspondences by matching point and pixel pat…
TripletVideo Object Segmentation using Supervoxel-Based Gerrymandering
Pixels operate locally. Superpixels have some potential to collect information across many pixels; supervoxels have more potential by implicitly operating across time. In this paper, we explore this well established noti…
ObjectSemantic SegmentationSuperpixelsUnsupervised Video Object Segmentation+3DiffuserCam: Lensless Single-exposure 3D Imaging
We demonstrate a compact and easy-to-build computational camera for single-shot 3D imaging. Our lensless system consists solely of a diffuser placed in front of a standard image sensor. Every point within the volumetric …
compressed sensingEvolving Modular Soft Robots without Explicit Inter-Module Communication using Local Self-Attention
Modularity in robotics holds great potential. In principle, modular robots can be disassembled and reassembled in different robots, and possibly perform new tasks. Nevertheless, actually exploiting modularity is yet an u…
Inductive BiasELBA: Learning by Asking for Embodied Visual Navigation and Task Completion
The research community has shown increasing interest in designing intelligent embodied agents that can assist humans in accomplishing tasks. Although there have been significant advancements in related vision-language be…
Question AnsweringVisual Navigation