S$^2$-Diffusion: Generalizing from Instance-level to Category-level Skills in Robot Manipulation
Recent advances in skill learning has propelled robot manipulation to new heights by enabling it to learn complex manipulation tasks from a practical number of demonstrations. However, these skills are often limited to the particular action, object, and environment \textit{instances} that are shown in the training data, and have trouble transferring to other instances of the same category. In this work we present an open-vocabulary Spatial-Semantic Diffusion policy (S$^2$-Diffusion) which enables generalization from instance-level training data to category-level, enabling skills to be transferable between instances of the same category. We show that functional aspects of skills can be captured via a promptable semantic module combined with a spatial representation. We further propose leveraging depth estimation networks to allow the use of only a single RGB camera. Our approach is evaluated and compared on a diverse number of robot manipulation tasks, both in simulation and in the real world. Our results show that S$^2$-Diffusion is invariant to changes in category-irrelevant factors as well as enables satisfying performance on other instances within the same category, even if it was not trained on that specific instance. Full videos of all real-world experiments are available in the supplementary material.
Code (0)
등록된 구현이 없습니다.
Tasks
Depth EstimationRobot ManipulationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Category-Level 6D Object Pose Estimation in the Wild: A Semi-Supervised Learning Approach and A New Dataset
6D object pose estimation is one of the fundamental problems in computer vision and robotics research. While a lot of recent efforts have been made on generalizing pose estimation to novel object instances within the sam…
6D Pose Estimation6D Pose Estimation using RGBObjectPose EstimationInstancePin: Instance-Addressable Layout-to-Image Diffusion via Coordinate Pinning
Layout-to-image diffusion models have achieved impressive semantic controllability by conditioning generation on category-level segmentation maps. However, such category-aligned control is not necessarily instance-addres…
Diffusion-Driven Self-Supervised Learning for Shape Reconstruction and Pose Estimation
Fully-supervised category-level pose estimation aims to determine the 6-DoF poses of unseen instances from known categories, requiring expensive mannual labeling costs. Recently, various self-supervised category-level po…
Pose EstimationSelf-Supervised LearningSCOPE: Semantic Conditioning for Sim2Real Category-Level Object Pose Estimation in Robotics
Object manipulation requires accurate object pose estimation. In open environments, robots encounter unknown objects, which requires semantic understanding in order to generalize both to known categories and beyond. To r…
Pose EstimationSemantic-Contact Fields for Category-Level Generalizable Tactile Tool Manipulation
Generalizing tool manipulation requires both semantic planning and precise physical control. Modern generalist robot policies, such as Vision-Language-Action (VLA) models, often lack the physical grounding required for c…