Local Policies Enable Zero-shot Long-horizon Manipulation

2024-10-29 · Murtaza Dalal, Min Liu, Walter Talbott, Chen Chen, Deepak Pathak, Jian Zhang, Ruslan Salakhutdinov

Sim2real for robotic manipulation is difficult due to the challenges of simulating complex contacts and generating realistic task distributions. To tackle the latter problem, we introduce ManipGen, which leverages a new class of policies for sim2real transfer: local policies. Locality enables a variety of appealing properties including invariances to absolute robot and object pose, skill ordering, and global scene configuration. We combine these policies with foundation models for vision, language and motion planning and demonstrate SOTA zero-shot performance of our method to Robosuite benchmark tasks in simulation (97%). We transfer our local policies from simulation to reality and observe they can solve unseen long-horizon manipulation tasks with up to 8 stages with significant pose, object and scene configuration variation. ManipGen outperforms SOTA approaches such as SayCan, OpenVLA, LLMTrajGen and VoxPoser across 50 real-world manipulation tasks by 36%, 76%, 62% and 60% respectively. Video results at https://mihdalal.github.io/manipgen/

📄 PDF Abstract BibTeX arXiv:2410.22332

Code (0)

등록된 구현이 없습니다.

Tasks

Motion Planning

Zero-shot Imitation Learning by Latent Topology Mapping

2026-05-08 · Maxwell J. Jacobson, Yexiang Xue arxiv

Imitation learning is effective for training agents when expert demonstrations are available, but collecting demonstrations for every complex task in an environment is costly. We study the long-horizon, goal-conditioned …

Foundation Policies with Hilbert Representations

2024-02-23 · Seohong Park, Tobias Kreiman, Sergey Levine

Unsupervised and self-supervised objectives, such as next token prediction, have enabled pre-training generalist models from large amounts of unlabeled data. In reinforcement learning (RL), however, finding a truly gener…

Reinforcement Learning (RL)Unsupervised Pre-training

TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learning

2025-10-01 · Marco Bagatella, Matteo Pirotta, Ahmed Touati, Alessandro Lazaric 외 arxiv

Latent prediction--where agents learn by predicting their own latents--has emerged as a powerful paradigm for training general representations in machine learning. In reinforcement learning (RL), this approach has been e…

Reinforcement Learning

Zero-Shot Function Encoder-Based Differentiable Predictive Control

2025-11-07 · Hassan Iqbal, Xingjian Li, Tyler Ingebrand, Adam Thorpe 외 arxiv

We introduce a differentiable framework for zero-shot adaptive control over parametric families of nonlinear dynamical systems. Our approach integrates a function encoder-based neural ODE (FE-NODE) for modeling system dy…

Self-Supervised Learning

A Unified Framework for Zero-Shot Reinforcement Learning

2025-10-23 · Jacopo Di Ventura, Jan Felix Kleuker, Aske Plaat, Thomas Moerland arxiv

Zero-shot reinforcement learning (RL) has emerged as a setting for developing general agents, capable of solving downstream tasks without additional training or planning at test-time. While conventional RL optimizes poli…