Learning 3D Dynamic Scene Representations for Robot Manipulation
3D scene representation for robot manipulation should capture three key object properties: permanency -- objects that become occluded over time continue to exist; amodal completeness -- objects have 3D occupancy, even if only partial observations are available; spatiotemporal continuity -- the movement of each object is continuous over space and time. In this paper, we introduce 3D Dynamic Scene Representation (DSR), a 3D volumetric scene representation that simultaneously discovers, tracks, reconstructs objects, and predicts their dynamics while capturing all three properties. We further propose DSR-Net, which learns to aggregate visual observations over multiple interactions to gradually build and refine DSR. Our model achieves state-of-the-art performance in modeling 3D scene dynamics with DSR on both simulated and real data. Combined with model predictive control, DSR-Net enables accurate planning in downstream robotic manipulation tasks such as planar pushing. Video is available at https://youtu.be/GQjYG3nQJ80.
Code (2)
Tasks
Model Predictive ControlRobot ManipulationSimilar Papers 제목 키워드 기반
GAF: Gaussian Action Field as a Dvnamic World Model for Robotic Mlanipulation
Accurate action inference is critical for vision-based robotic manipulation. Existing approaches typically follow either a Vision-to-Action (V-A) paradigm, predicting actions directly from visual inputs, or a Vision-to-3…
3DGSToken Bottleneck: One Token to Remember Dynamics
Deriving compact and temporally aware visual representations from dynamic scenes is essential for successful execution of sequential scene understanding tasks such as visual tracking and robotic manipulation. In this pap…
Self-Supervised LearningScene UnderstandingRobot ManipulationVisual TrackingFlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulation
This paper investigates training better visual world models for robot manipulation, i.e., models that can predict future visual observations by conditioning on past frames and robot actions. Specifically, we consider wor…
Robot ManipulationSemantic SimilaritySemantic Textual SimilarityVideo Prediction3D Neural Scene Representations for Visuomotor Control
Humans have a strong intuitive understanding of the 3D environment around us. The mental model of the physics in our brain applies to objects of different materials and enables us to perform a wide range of manipulation …
Contrastive LearningFuture predictionNeRFNovel View SynthesisManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation
Performing language-conditioned robotic manipulation tasks in unstructured environments is highly demanded for general intelligent robots. Conventional robotic manipulation methods usually learn semantic representation o…
Simulated Gaussian Manipulation