Visual Robot Task Planning
Prospection, the act of predicting the consequences of many possible futures, is intrinsic to human planning and action, and may even be at the root of consciousness. Surprisingly, this idea has been explored comparatively little in robotics. In this work, we propose a neural network architecture and associated planning algorithm that (1) learns a representation of the world useful for generating prospective futures after the application of high-level actions, (2) uses this generative model to simulate the result of sequences of high-level actions in a variety of environments, and (3) uses this same representation to evaluate these actions and perform tree search to find a sequence of high-level actions in a new environment. Models are trained via imitation learning on a variety of domains, including navigation, pick-and-place, and a surgical robotics task. Our approach allows us to visualize intermediate motion goals and learn to plan complex activity from visual information.
Code (1)
Tasks
Imitation LearningRobot Task PlanningTask PlanningSimilar Papers 제목 키워드 기반
Seeing What Matters: Visual Cue Guided Video Planning for Generalizable Robot Navigation
Generative video models can serve as a promising backbone for robot navigation by predicting future observations as video plans. Recent approaches often condition video planning on short-horizon guidance and recover geom…
Robot NavigationGameVLM: A Decision-making Framework for Robotic Task Planning Based on Visual Language Models and Zero-sum Games
With their prominent scene understanding and reasoning capabilities, pre-trained visual-language models (VLMs) such as GPT-4V have attracted increasing attention in robotic task planning. Compared with traditional task p…
Code GenerationDecision MakingHallucinationScene Understanding+1Wonderful Team: Zero-Shot Physical Task Planning with Visual LLMs
We introduce Wonderful Team, a multi-agent Vision Large Language Model (VLLM) framework for executing high-level robotic planning in a zero-shot regime. In our context, zero-shot high-level planning means that for a nove…
Action GenerationLarge Language ModelTask PlanningVisual ReasoningLook Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning
In this study, we are interested in imbuing robots with the capability of physically-grounded task planning. Recent advancements have shown that large language models (LLMs) possess extensive knowledge useful in robotic …
Task PlanningAssessing the Value of Visual Input: A Benchmark of Multimodal Large Language Models for Robotic Path Planning
Large Language Models (LLMs) show potential for enhancing robotic path planning. This paper assesses visual input's utility for multimodal LLMs in such tasks via a comprehensive benchmark. We evaluated 15 multimodal LLMs…
Spatial Reasoning