Translating Natural Language Instructions to Computer Programs for Robot Manipulation
It is highly desirable for robots that work alongside humans to be able to understand instructions in natural language. Existing language conditioned imitation learning models directly predict the actuator commands from the image observation and the instruction text. Rather than directly predicting actuator commands, we propose translating the natural language instruction to a Python function which queries the scene by accessing the output of the object detector and controls the robot to perform the specified task. This enables the use of non-differentiable modules such as a constraint solver when computing commands to the robot. Moreover, the labels in this setup are significantly more informative computer programs that capture the intent of the expert rather than teleoperated demonstrations. We show that the proposed method performs better than training a neural network to directly predict the robot actions.
Code (0)
등록된 구현이 없습니다.
Tasks
Imitation LearningRobot ManipulationSimilar Papers 제목 키워드 기반
LLMs can generate robotic scripts from goal-oriented instructions in biological laboratory automation
The use of laboratory automation by all researchers may substantially accelerate scientific activities by humans, including those in the life sciences. However, computer programs to operate robots should be written to im…
Communicating Natural Programs to Humans and Machines
The Abstraction and Reasoning Corpus (ARC) is a set of procedural tasks that tests an agent's ability to flexibly solve novel problems. While most ARC tasks are easy for humans, they are challenging for state-of-the-art …
ARCProgram SynthesisTranslating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation
We propose an end-to-end deep learning model for translating free-form natural language instructions to a high-level plan for behavioral robot navigation. We use attention models to connect information from both the user…
FormRobot NavigationTranslationTranslating Natural Language Instructions for Behavioral Robot Navigation with a Multi-Head Attention Mechanism
We propose a multi-head attention mechanism as a blending layer in a neural network model that translates natural language to a high level behavioral language for indoor robot navigation. We follow the framework establis…
Robot NavigationProgram Language Translation Using a Grammar-Driven Tree-to-Tree Model
The task of translating between programming languages differs from the challenge of translating natural languages in that programming languages are designed with a far more rigid set of structural and grammatical rules. …
DecoderTranslation