Multi-step manipulation task and motion planning guided by video demonstration
This work aims to leverage instructional video to solve complex multi-step task-and-motion planning tasks in robotics. Towards this goal, we propose an extension of the well-established Rapidly-Exploring Random Tree (RRT) planner, which simultaneously grows multiple trees around grasp and release states extracted from the guiding video. Our key novelty lies in combining contact states and 3D object poses extracted from the guiding video with a traditional planning algorithm that allows us to solve tasks with sequential dependencies, for example, if an object needs to be placed at a specific location to be grasped later. We also investigate the generalization capabilities of our approach to go beyond the scene depicted in the instructional video. To demonstrate the benefits of the proposed video-guided planning approach, we design a new benchmark with three challenging tasks: (I) 3D re-arrangement of multiple objects between a table and a shelf, (ii) multi-step transfer of an object through a tunnel, and (iii) transferring objects using a tray similar to a waiter transfers dishes. We demonstrate the effectiveness of our planning algorithm on several robots, including the Franka Emika Panda and the KUKA KMR iiwa. For a seamless transfer of the obtained plans to the real robot, we develop a trajectory refinement approach formulated as an optimal control problem (OCP).
Code (0)
등록된 구현이 없습니다.
Tasks
Motion PlanningTask and Motion PlanningSimilar Papers 제목 키워드 기반
Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation
The fundamental challenge of planning for multi-step manipulation is to find effective and plausible action sequences that lead to the task goal. We present Cascaded Variational Inference (CAVIN) Planner, a model-based m…
Variational InferenceIntegration of Robot and Scene Kinematics for Sequential Mobile Manipulation Planning
We present a Sequential Mobile Manipulation Planning (SMMP) framework that can solve long-horizon multi-step mobile manipulation tasks with coordinated whole-body motion, even when interacting with articulated objects. B…
DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observability
Generative models such as diffusion models, excel at capturing high-dimensional distributions with diverse input modalities, e.g. robot trajectories, but are less effective at multi-step constraint reasoning. Task and Mo…
Motion PlanningRobot ManipulationTask and Motion PlanningAccelerating Integrated Task and Motion Planning with Neural Feasibility Checking
As robots play an increasingly important role in the industrial, the expectations about their applications for everyday living tasks are getting higher. Robots need to perform long-horizon tasks that consist of several s…
Motion PlanningTask and Motion PlanningLong-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances
We present a strategy for designing and building very general robot manipulation systems involving the integration of a general-purpose task-and-motion planner with engineered and learned perception modules that estimate…
Grasp GenerationMotion PlanningRobot ManipulationTask and Motion Planning