RGB-Only Reconstruction of Tabletop Scenes for Collision-Free Manipulator Control
We present a system for collision-free control of a robot manipulator that uses only RGB views of the world. Perceptual input of a tabletop scene is provided by multiple images of an RGB camera (without depth) that is either handheld or mounted on the robot end effector. A NeRF-like process is used to reconstruct the 3D geometry of the scene, from which the Euclidean full signed distance function (ESDF) is computed. A model predictive control algorithm is then used to control the manipulator to reach a desired pose while avoiding obstacles in the ESDF. We show results on a real dataset collected and annotated in our lab.
Code (0)
등록된 구현이 없습니다.
Tasks
3D geometryModel Predictive ControlNeRFSimilar Papers 제목 키워드 기반
TabletopGen: Tabletop Scene Generation and Interactive Simulation for Robotic Manipulation
Simulation provides a low-cost, scalable pathway to large-scale robotic manipulation data collection. However, existing 3D scene generation methods can rarely be applied directly to manipulation data synthesis, as their …
Scene GenerationObject Rearrangement Using Learned Implicit Collision Functions
Robotic object rearrangement combines the skills of picking and placing objects. When object models are unavailable, typical collision-checking models may be unable to predict collisions in partial point clouds with occl…
ObjectObject RearrangementSTABLE: Simulation-Ready Tabletop Layout Generation via a Semantics-Physics Dual System
Generating simulation-ready tabletop scenes from task instructions is an intriguing and promising research direction in the field of Embodied AI. However, existing task-to-scene generation methods rely exclusively on lar…
Spatial ReasoningScene GenerationPhyScene3D: Physically Consistent Interactive 3D Tabletop Scene Generation
Generating physically consistent 3D tabletop scenes is a fundamental yet underexplored problem for interactive and generalist robotic learning. The challenge stems from dense object hierarchies and irregular affordances.…
Scene GenerationTO-Scene: A Large-scale Dataset for Understanding 3D Tabletop Scenes
Many basic indoor activities such as eating or writing are always conducted upon different tabletops (e.g., coffee tables, writing desks). It is indispensable to understanding tabletop scenes in 3D indoor scene parsing a…
3D Semantic Segmentationobject-detectionObject DetectionScene Parsing+1