Generalizing MLPs With Dropouts, Batch Normalization, and Skip Connections
A multilayer perceptron (MLP) is typically made of multiple fully connected layers with nonlinear activation functions. There have been several approaches to make them better (e.g. faster convergence, better convergence limit, etc.). But the researches lack structured ways to test them. We test different MLP architectures by carrying out the experiments on the age and gender datasets. We empirically show that by whitening inputs before every linear layer and adding skip connections, our proposed MLP architecture can result in better performance. Since the whitening process includes dropouts, it can also be used to approximate Bayesian inference. We have open sourced our code, and released models and docker images at https://github.com/tae898/age-gender/
Code (1)
Tasks
Age And Gender ClassificationAge EstimationGender PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Dynamical Isometry for Residual Networks
The training success, training speed and generalization ability of neural networks rely crucially on the choice of random parameter initialization. It has been shown for multiple architectures that initial dynamical isom…
DiversityCharacterizing Well-Behaved vs. Pathological Deep Neural Networks
We introduce a novel approach, requiring only mild assumptions, for the characterization of deep neural networks at initialization. Our approach applies both to fully-connected and convolutional networks and easily incor…
Batch Normalization has Multiple Benefits: An Empirical Study on Residual Networks
Many state of the art models rely on two architectural innovations; skip connections and batch normalization. However batch normalization has a number of limitations. It breaks the independence between training examples …
Skip-connection and batch-normalization improve data separation ability
The ResNet and the batch-normalization (BN) achieved high performance even when only a few labeled data are available. However, the reasons for its high performance are unclear. To clear the reasons, we analyzed the effe…
Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
Batch normalization dramatically increases the largest trainable depth of residual networks, and this benefit has been crucial to the empirical success of deep residual networks on a wide range of benchmarks. We show tha…