Object-Centric World Model for Language-Guided Manipulation

2025-03-08 · Youngjoon Jeong, Junha Chun, Soonwoo Cha, Taesup Kim

A world model is essential for an agent to predict the future and plan in domains such as autonomous driving and robotics. To achieve this, recent advancements have focused on video generation, which has gained significant attention due to the impressive success of diffusion models. However, these models require substantial computational resources. To address these challenges, we propose a world model leveraging object-centric representation space using slot attention, guided by language instructions. Our model perceives the current state as an object-centric representation and predicts future states in this representation space conditioned on natural language instructions. This approach results in a more compact and computationally efficient model compared to diffusion-based generative alternatives. Furthermore, it flexibly predicts future states based on language instructions, and offers a significant advantage in manipulation tasks where object recognition is crucial. In this paper, we demonstrate that our latent predictive world model surpasses generative world models in visuo-linguo-motor control tasks, achieving superior sample and computation efficiency. We also investigate the generalization performance of the proposed method and explore various strategies for predicting actions using object-centric representations.

📄 PDF Abstract BibTeX arXiv:2503.06170

Code (0)

등록된 구현이 없습니다.

Tasks

Autonomous DrivingmodelObjectObject Recognition Video Generation

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

EC-Flow: Enabling Versatile Robotic Manipulation from Action-Unlabeled Videos via Embodiment-Centric Flow

2025-07-08 · Yixiang Chen, Peiyan Li, Yan Huang, Jiabing Yang 외

Current language-guided robotic manipulation systems often require low-level action-labeled datasets for imitation learning. While object-centric flow prediction methods mitigate this issue, they remain limited to scenar…

Deformable Object Manipulation Imitation Learning Object

UniHM: Unified Dexterous Hand Manipulation with Vision Language Model

2026-02-28 · Zhenhao Zhang, Jiaxin Liu, Ye Shi, Jingya Wang arxiv

Planning physically feasible dexterous hand manipulation is a central challenge in robotic manipulation and Embodied AI. Prior work typically relies on object-centric cues or precise hand-object interaction sequences, fo…

SAM3D-Guided Object-Centric Representation Alignment for Vision-Language-Action Models

2026-07-28 · Zonghe Liu, Shanyuan Jie, Xiaoquan Sun, Chen Cao 외 arxiv

Vision-Language-Action (VLA) models have shown strong potential for general robot manipulation, but most existing models rely on 2D visual-language backbones and lack fine-grained 3D understanding of target objects, espe…

Object Recognition Robot Manipulation Point Clouds

FOCUS: Object-Centric World Models for Robotics Manipulation

2023-07-05 · Stefano Ferraro, Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt

Understanding the world in terms of objects and the possible interplays with them is an important cognition ability, especially in robotics manipulation, where many tasks require robot-object interactions. However, learn…

Object

Wh0: Generative World Models as Scalable Sources of Egocentric Human Hand Manipulation Data

2026-06-20 · Yangtao Chen, Zixuan Chen, Peiyang Wang, Yong-Lu Li 외 arxiv

Scaling dexterous manipulation requires generalization across objects, scenes, and tasks, yet existing data sources face a trade-off between scale and scene/embodiment alignment: teleoperation data is well aligned with r…