Lifting AutoEncoders: Unsupervised Learning of a Fully-Disentangled 3D Morphable Model using Deep Non-Rigid Structure from Motion
In this work we introduce Lifting Autoencoders, a generative 3D surface-based model of object categories. We bring together ideas from non-rigid structure from motion, image formation, and morphable models to learn a controllable, geometric model of 3D categories in an entirely unsupervised manner from an unstructured set of images. We exploit the 3D geometric nature of our model and use normal information to disentangle appearance into illumination, shading and albedo. We further use weak supervision to disentangle the non-rigid shape variability of human faces into identity and expression. We combine the 3D representation with a differentiable renderer to generate RGB images and append an adversarially trained refinement network to obtain sharp, photorealistic image reconstruction results. The learned generative model can be controlled in terms of interpretable geometry and appearance factors, allowing us to perform photorealistic image manipulation of identity, expression, 3D pose, and illumination properties.
Code (0)
등록된 구현이 없습니다.
Tasks
Image ManipulationImage ReconstructionSimilar Papers 제목 키워드 기반
MOST-GAN: 3D Morphable StyleGAN for Disentangled Face Image Manipulation
Recent advances in generative adversarial networks (GANs) have led to remarkable achievements in face image synthesis. While methods that use style-based GANs can generate strikingly photorealistic face images, it is oft…
DisentanglementImage GenerationImage ManipulationUnsupervised Model Selection for Variational Disentangled Representation Learning
Disentangled representations have recently been shown to improve fairness, data efficiency and generalisation in simple supervised and reinforcement learning tasks. To extend the benefits of disentangled representations …
AttributeDisentanglementFairnessmodel+3i3DMM: Deep Implicit 3D Morphable Model of Human Heads
We present the first deep implicit 3D morphable model (i3DMM) of full heads. Unlike earlier morphable face models it not only captures identity-specific geometry, texture, and expressions of the frontal face, but also mo…
Evaluating unsupervised disentangled representation learning for genomic discovery and disease risk prediction
High-dimensional clinical data have become invaluable resources for genetic studies, due to their accessibility in biobank-scale datasets and the development of high performance modeling techniques especially using deep …
Representation LearningUnsupervised Disentanglement without Autoencoding: Pitfalls and Future Directions
Disentangled visual representations have largely been studied with generative models such as Variational AutoEncoders (VAEs). While prior work has focused on generative methods for disentangled representation learning, t…
Contrastive LearningDisentanglementRepresentation LearningSensitivity