3D Morphable Models as Spatial Transformer Networks

2017-08-23 · Anil Bas, Patrik Huber, William A. P. Smith, Muhammad Awais, Josef Kittler

In this paper, we show how a 3D Morphable Model (i.e. a statistical model of the 3D shape of a class of objects such as faces) can be used to spatially transform input data as a module (a 3DMM-STN) within a convolutional neural network. This is an extension of the original spatial transformer network in that we are able to interpret and normalise 3D pose changes and self-occlusions. The trained localisation part of the network is independently useful since it learns to fit a 3D morphable model to a single image. We show that the localiser can be trained using only simple geometric loss functions on a relatively small dataset yet is able to perform robust normalisation on highly uncontrolled images including occlusion, self-occlusion and large pose changes.

📄 PDF Abstract BibTeX arXiv:1708.07199

Code (1)

anilbas/3DMMasSTN 공식 구현

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Absolute Position Encodings Absolute Position Encodings are a type of position embeddings for [Transformer-based models] where positional encodings are…

Position-Wise Feed-Forward Layer 설명 없음

Spatial Transformer A Spatial Transformer is an image model block that explicitly allows the spatial manipulation of data within a [convolutional neural…

Residual Connection 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Label Smoothing Label Smoothing is a regularization technique that introduces noise for the labels. This accounts for the fact that datasets may have mistakes in them, so maximizing the…

Learning a 3D Morphable Face Reflectance Model from Low-cost Data

2023-03-21 · CVPR 2023 1 · Yuxuan Han, Zhibo Wang, Feng Xu

Modeling non-Lambertian effects such as facial specularity leads to a more realistic 3D Morphable Face Model. Existing works build parametric models for diffuse and specular albedo using Light Stage data. However, only d…

Face Model Inverse Rendering

Morphable Face Models - An Open Framework

2017-09-25 · Thomas Gerig, Andreas Morel-Forster, Clemens Blumer, Bernhard Egger 외

In this paper, we present a novel open-source pipeline for face registration based on Gaussian processes as well as an application to face image analysis. Non-rigid registration of faces is significant for many applicati…

Face Model Gaussian Processes

Efficient Global Illumination for Morphable Models

2017-10-01 · ICCV 2017 10 · Andreas Schneider, Sandro Schonborn, Lavrenti Frobeen, Bernhard Egger 외

We propose an efficient self-shadowing illumination model for Morphable Models. Simulating self-shadowing with ray casting is computationally expensive which makes them impractical in Analysis-by-Synthesis methods for ob…

Object Reconstruction

CSANet: Channel Spatial Attention Network for Robust 3D Face Alignment and Reconstruction

2024-05-30 · Yilin Liu, Xuezhou Guo, Xinqi Wang, Fangzhou Du

Our project proposes an end-to-end 3D face alignment and reconstruction network. The backbone of our model is built by Bottle-Neck structure via Depth-wise Separable Convolution. We integrate Coordinate Attention mechani…

3D Face Alignment Face Alignment

Building 3D Morphable Models from a Single Scan

2020-11-24 · Skylar Sutherland, Bernhard Egger, Joshua Tenenbaum

We propose a method for constructing generative models of 3D objects from a single 3D mesh. Our method produces a 3D morphable model that represents shape and albedo in terms of Gaussian processes. We define the shape de…

Face Recognition Gaussian Processes Inverse Rendering