Zero-shot Reconstruction of In-Scene Object Manipulation from Video

2025-12-22 · Dixuan Lin, Tianyou Wang, Zhuoyang Pan, Yufu Wang, Lingjie Liu, Kostas Daniilidis arxiv

We build the first system to address the problem of reconstructing in-scene object manipulation from a monocular RGB video. It is challenging due to ill-posed scene reconstruction, ambiguous hand-object depth, and the need for physically plausible interactions. Existing methods operate in hand centric coordinates and ignore the scene, hindering metric accuracy and practical use. In our method, we first use data-driven foundation models to initialize the core components, including the object mesh and poses, the scene point cloud, and the hand poses. We then apply a two-stage optimization that recovers a complete hand-object motion from grasping to interaction, which remains consistent with the scene information observed in the input video.

📄 PDF Abstract BibTeX arXiv:2512.19684

Code (0)

등록된 구현이 없습니다.

ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors

2026-03-04 · Zihao Huang, Tianqi Liu, Zhaoxi Chen, Shaocong Xu 외 arxiv

Synthesizing physically plausible articulated human-object interactions (HOI) without 3D/4D supervision remains a fundamental challenge. While recent zero-shot approaches leverage video diffusion models to synthesize hum…

Inverse Rendering

GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulation

2025-10-23 · Guangqi Jiang, Haoran Chang, Ri-Zhao Qiu, Yutong Liang 외 arxiv

This paper presents GSWorld, a robust, photo-realistic simulator for robotics manipulation that combines 3D Gaussian Splatting with physics engines. Our framework advocates "closing the loop" of developing manipulation p…

Reinforcement Learning Robot Manipulation

Gaussian Object Carver: Object-Compositional Gaussian Splatting with surfaces completion

2024-12-03 · Liu Liu, Xinjie Wang, Jiaxiong Qiu, Tianwei Lin 외

3D scene reconstruction is a foundational problem in computer vision. Despite recent advancements in Neural Implicit Representations (NIR), existing methods often lack editability and compositional flexibility, limiting …

3D Scene Reconstruction Object

SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

2026-06-26 · Nadun Ranawaka, Josiah Wong, Wei-Lin Pai, Wei-Teng Chu 외 arxiv

Training and evaluating robot policies in the real world is costly and difficult to scale. We introduce SimFoundry, a modular and automated system for zero-shot real-to-sim scene construction from a video. SimFoundry gen…

Scene Generation

Differentiable Inverse Graphics for Zero-shot Scene Reconstruction and Robot Grasping

2026-02-04 · Octavio Arriaga, Proneet Sharma, Jichen Guo, Marc Otto 외 arxiv

Operating effectively in novel real-world environments requires robotic systems to estimate and interact with previously unseen objects. Current state-of-the-art models address this challenge by using large amounts of tr…

Pose Estimation