Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation

2025-01-08 · Senwei Xie, Hongyu Wang, Zhanqi Xiao, Ruiping Wang, Xilin Chen

Zero-shot generalization across various robots, tasks and environments remains a significant challenge in robotic manipulation. Policy code generation methods use executable code to connect high-level task descriptions and low-level action sequences, leveraging the generalization capabilities of large language models and atomic skill libraries. In this work, we propose Robotic Programmer (RoboPro), a robotic foundation model, enabling the capability of perceiving visual information and following free-form instructions to perform robotic manipulation with policy code in a zero-shot manner. To address low efficiency and high cost in collecting runtime code data for robotic tasks, we devise Video2Code to synthesize executable code from extensive videos in-the-wild with off-the-shelf vision-language model and code-domain large language model. Extensive experiments show that RoboPro achieves the state-of-the-art zero-shot performance on robotic manipulation in both simulators and real-world environments. Specifically, the zero-shot success rate of RoboPro on RLBench surpasses the state-of-the-art model GPT-4o by 11.6%, which is even comparable to a strong supervised training baseline. Furthermore, RoboPro is robust to variations on API formats and skill sets.

📄 PDF Abstract BibTeX arXiv:2501.04268

Code (0)

등록된 구현이 없습니다.

Tasks

Code GenerationLanguage ModelingLanguage ModellingLarge Language ModelZero-shot Generalization

Cross-Domain Demo-to-Code via Neurosymbolic Counterfactual Reasoning

2026-03-19 · Jooyoung Kim, Wonje Choi, Younguk Song, Honguk Woo arxiv

Recent advances in Vision-Language Models (VLMs) have enabled video-instructed robotic programming, allowing agents to interpret video demonstrations and generate executable control code. We formulate video-instructed ro…

Domain Adaptation

Octo: An Open-Source Generalist Robot Policy

2024-05-20 · Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch 외

Large policies pretrained on diverse robot datasets have the potential to transform robotic learning: instead of training new policies from scratch, such generalist robot policies may be finetuned with only a little in-d…

Robot Manipulation

One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos

2024-09-29 · Zechen Bai, Tong He, Haiyang Mei, Pichao Wang 외

We introduce VideoLISA, a video-based multimodal large language model designed to tackle the problem of language-instructed reasoning segmentation in videos. Leveraging the reasoning capabilities and world knowledge of l…

All Image Segmentation Language Modeling Language Modelling+11

Automatic Robotic Development through Collaborative Framework by Large Language Models

2024-02-06 · Zhirong Luan, Yujun Lai, Rundong Huang, Xiaruiqi Lan 외

Despite the remarkable code generation abilities of large language models LLMs, they still face challenges in complex task handling. Robot development, a highly intricate field, inherently demands human involvement in ta…

Code Generation

Training a Conditioned Video Game Agent on a VLM Annotated Dataset

2026-08-06 · Katrin Schmid, Iuri Frosio arxiv

Reinforcement Learning (RL) is a powerful but far from easy-to-use technique for policy learning. In the specific case of video games, access to the game engine is required to get rewards for training (e.g. to collect re…

Reinforcement Learning Offline RL