Differentiable Robot Rendering
Vision foundation models trained on massive amounts of visual data have shown unprecedented reasoning and planning skills in open-world settings. A key challenge in applying them to robotic tasks is the modality gap between visual data and action data. We introduce differentiable robot rendering, a method allowing the visual appearance of a robot body to be directly differentiable with respect to its control parameters. Our model integrates a kinematics-aware deformable model and Gaussians Splatting and is compatible with any robot form factors and degrees of freedom. We demonstrate its capability and usage in applications including reconstruction of robot poses from images and controlling robots through vision language models. Quantitative and qualitative results show that our differentiable rendering model provides effective gradients for robotic control directly from pixels, setting the foundation for the future applications of vision foundation models in robotics.
Code (1)
Similar Papers 제목 키워드 기반
Prof. Robot: Differentiable Robot Rendering Without Static and Self-Collisions
Differentiable rendering has gained significant attention in the field of robotics, with differentiable robot rendering emerging as an effective paradigm for learning robotic actions from image-space supervision. However…
DiffGen: Robot Demonstration Generation via Differentiable Physics Simulation, Differentiable Rendering, and Vision-Language Model
Generating robot demonstrations through simulation is widely recognized as an effective way to scale up robot data. Previous work often trained reinforcement learning agents to generate expert policies, but this approach…
Language ModelingLanguage ModellingRobot ManipulationOne-Shot Real-to-Sim via End-to-End Differentiable Simulation and Rendering
Identifying predictive world models for robots in novel environments from sparse online observations is essential for robot task planning and execution in novel environments. However, existing methods that leverage diffe…
BenchmarkingObjectRobot Task PlanningTask PlanningStreamlining stereo differentiable rendering for marker-free real-time tracking of surgical robots
Purpose: Marker-based tracking of surgical robots is occlusion-prone in cluttered operating rooms. We evaluate stereo differentiable rendering for marker-free, real-time robot pose tracking, potentially improving safety,…
Pose EstimationPose TrackingDiffPhysCam: Differentiable Physics-Based Camera Simulation for Inverse Rendering and Embodied AI
Generating synthetic images that closely mimic those from real cameras is instrumental in training visual models and enabling end-to-end visuomotor learning. We introduce DiffPhysCam, a differentiable camera simulator de…
Inverse Rendering