4DComplete: Non-Rigid Motion Estimation Beyond the Observable Surface
Tracking non-rigidly deforming scenes using range sensors has numerous applications including computer vision, AR/VR, and robotics. However, due to occlusions and physical limitations of range sensors, existing methods only handle the visible surface, thus causing discontinuities and incompleteness in the motion field. To this end, we introduce 4DComplete, a novel data-driven approach that estimates the non-rigid motion for the unobserved geometry. 4DComplete takes as input a partial shape and motion observation, extracts 4D time-space embedding, and jointly infers the missing geometry and motion field using a sparse fully-convolutional network. For network training, we constructed a large-scale synthetic dataset called DeformingThings4D, which consists of 1972 animation sequences spanning 31 different animals or humanoid categories with dense 4D annotation. Experiments show that 4DComplete 1) reconstructs high-resolution volumetric shape and motion field from a partial observation, 2) learns an entangled 4D feature representation that benefits both shape and motion estimation, 3) yields more accurate and natural deformation than classic non-rigid priors such as As-Rigid-As-Possible (ARAP) deformation, and 4) generalizes well to unseen objects in real-world sequences.
Code (1)
Tasks
Motion EstimationSimilar Papers 제목 키워드 기반
EffiScene: Efficient Per-Pixel Rigidity Inference for Unsupervised Joint Learning of Optical Flow, Depth, Camera Pose and Motion Segmentation
This paper addresses the challenging unsupervised scene flow estimation problem by jointly learning four low-level vision sub-tasks: optical flow $\textbf{F}$, stereo-depth $\textbf{D}$, camera pose $\textbf{P}$ and moti…
Depth EstimationMotion SegmentationOptical Flow EstimationScene Flow Estimation+1Koopman Operator Based Modeling and Control of Rigid Body Motion Represented by Dual Quaternions
In this paper, we systematically derive a finite set of Koopman based observables to construct a lifted linear state space model that describes the rigid body dynamics based on the dual quaternion representation. In gene…
Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation
Estimation of 3D motion in a dynamic scene from a temporal pair of images is a core task in many scene understanding problems. In real world applications, a dynamic scene is commonly captured by a moving camera (i.e., pa…
Optical Flow EstimationScene Flow EstimationScene UnderstandingImage-Space Gridding for Nonrigid Motion-Corrected MR Image Reconstruction
Motion remains a major challenge in magnetic resonance (MR) imaging, particularly in free-breathing cardiac MR imaging, where data are acquired over multiple heartbeats at varying respiratory phases. We adopt a model-bas…
Image ReconstructionMotion EstimationLearning to Segment Rigid Motions from Two Frames
Appearance-based detectors achieve remarkable performance on common scenes, but tend to fail for scenarios lack of training data. Geometric motion segmentation algorithms, however, generalize to novel scenes, but have ye…
Motion SegmentationScene Flow EstimationSegmentationVocal Bursts Valence Prediction