Pre-training Auto-regressive Robotic Models with 4D Representations

2025-02-18 · Dantong Niu, Yuvan Sharma, Haoru Xue, Giscard Biamby, Junyi Zhang, Ziteng Ji, Trevor Darrell, Roei Herzig

Foundation models pre-trained on massive unlabeled datasets have revolutionized natural language and computer vision, exhibiting remarkable generalization capabilities, thus highlighting the importance of pre-training. Yet, efforts in robotics have struggled to achieve similar success, limited by either the need for costly robotic annotations or the lack of representations that effectively model the physical world. In this paper, we introduce ARM4R, an Auto-regressive Robotic Model that leverages low-level 4D Representations learned from human video data to yield a better pre-trained robotic model. Specifically, we focus on utilizing 3D point tracking representations from videos derived by lifting 2D representations into 3D space via monocular depth estimation across time. These 4D representations maintain a shared geometric structure between the points and robot state representations up to a linear transformation, enabling efficient transfer learning from human video data to low-level robotic control. Our experiments show that ARM4R can transfer efficiently from human video data to robotics and consistently improves performance on tasks across various robot environments and configurations.

📄 PDF Abstract BibTeX arXiv:2502.13142

Code (0)

등록된 구현이 없습니다.

Tasks

Depth Estimation Monocular Depth Estimation Point Tracking Transfer Learning

Methods 이 논문이 사용한 방법론

Focus 설명 없음

An Empirical Study of Autoregressive Pre-training from Videos

2025-01-09 · Jathushan Rajasegaran, Ilija Radosavovic, Rahul Ravishankar, Yossi Gandelsman 외

We empirically study autoregressive pre-training from videos. To perform our study, we construct a series of autoregressive video models, called Toto. We treat videos as sequences of visual tokens and train transformer m…

Object Tracking Video Classification

Physical Autoregressive Model for Robotic Manipulation without Action Pretraining

2025-08-13 · Zijian Song, Sihan Qin, Tianshui Chen, Liang Lin 외 arxiv

The scarcity of manipulation data has motivated the use of pretrained large models from other modalities in robotics. In this work, we build upon autoregressive video generation models to propose a Physical Autoregressiv…

Video Generation Video Prediction

CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Prediction

2024-12-09 · Zhefei Gong, Pengxiang Ding, Shangke Lyu, Siteng Huang 외

In robotic visuomotor policy learning, diffusion-based models have achieved significant success in improving the accuracy of action trajectory generation compared to traditional autoregressive models. However, they suffe…

Action Generation Denoising

Quantization-Free Autoregressive Action Transformer

2025-03-18 · Ziyad Sheebaelhamd, Michael Tschannen, Michael Muehlebach, Claire Vernade

Current transformer-based imitation learning approaches introduce discrete action representations and train an autoregressive transformer decoder on the resulting latent code. However, the initial quantization breaks the…

Imitation Learning Quantization Sequential Decision Making

RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulation

2026-08-20 · Shaoxuan Wang, Guangting Zheng, Rui Huang, Zhipeng Tang 외 arxiv

Offline reinforcement learning improves robotic policies using previously collected data without further environment interaction. Yet prevalent diffusion- and flow-matching robot policies lack tractable likelihoods, limi…

Reinforcement Learning Offline RL