Online Adaptation for Consistent Mesh Reconstruction in the Wild
This paper presents an algorithm to reconstruct temporally consistent 3D meshes of deformable object instances from videos in the wild. Without requiring annotations of 3D mesh, 2D keypoints, or camera pose for each video frame, we pose video-based reconstruction as a self-supervised online adaptation problem applied to any incoming test video. We first learn a category-specific 3D reconstruction model from a collection of single-view images of the same category that jointly predicts the shape, texture, and camera pose of an image. Then, at inference time, we adapt the model to a test video over time using self-supervised regularization terms that exploit temporal consistency of an object instance to enforce that all reconstructed meshes share a common texture map, a base shape, as well as parts. We demonstrate that our algorithm recovers temporally consistent and reliable 3D structures from videos of non-rigid objects including those of animals captured in the wild -- an extremely challenging task rarely addressed before.
Code (0)
등록된 구현이 없습니다.
Tasks
3D ReconstructionSimilar Papers 제목 키워드 기반
Monocular 3D Object Reconstruction with GAN Inversion
Recovering a textured 3D mesh from a monocular image is highly challenging, particularly for in-the-wild objects that lack 3D ground truths. In this work, we present MeshInversion, a novel framework to improve the recons…
3D Object ReconstructionObjectObject ReconstructionWeakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild
We introduce a simple and effective network architecture for monocular 3D hand pose estimation consisting of an image encoder followed by a mesh convolutional decoder that is trained through a direct 3D hand mesh reconst…
3D Hand Pose EstimationDecoderHand Pose EstimationPose EstimationA Large-Scale 3D Face Mesh Video Dataset via Neural Re-parameterized Optimization
We propose NeuFace, a 3D face mesh pseudo annotation method on videos via neural re-parameterized optimization. Despite the huge progress in 3D face reconstruction methods, generating reliable 3D face labels for in-the-w…
3D Face ReconstructionDiversityFace ReconstructionBilevel Online Adaptation for Out-of-Domain Human Mesh Reconstruction
This paper considers a new problem of adapting a pre-trained model of human mesh reconstruction to out-of-domain streaming videos. However, most previous methods based on the parametric SMPL model \cite{loper2015smpl} un…
3D Human Pose EstimationMesh Density Adaptation for Template-based Shape Reconstruction
In 3D shape reconstruction based on template mesh deformation, a regularization, such as smoothness energy, is employed to guide the reconstruction into a desirable direction. In this paper, we highlight an often overloo…
3D Shape ReconstructionInverse Rendering