An Effective and Efficient Initialization Scheme for Training Multi-layer Feedforward Neural Networks
Network initialization is the first and critical step for training neural networks. In this paper, we propose a novel network initialization scheme based on the celebrated Stein's identity. By viewing multi-layer feedforward neural networks as cascades of multi-index models, the projection weights to the first hidden layer are initialized using eigenvectors of the cross-moment matrix between the input's second-order score function and the response. The input data is then forward propagated to the next layer and such a procedure can be repeated until all the hidden layers are initialized. Finally, the weights for the output layer are initialized by generalized linear modeling. Such a proposed SteinGLM method is shown through extensive numerical results to be much faster and more accurate than other popular methods commonly used for training neural networks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Depth-Aware Initialization for Stable and Efficient Neural Network Training
In past few years, various initialization schemes have been proposed. These schemes are glorot initialization, He initialization, initialization using orthogonal matrix, random walk method for initialization. Some of the…
Multilevel Initialization for Layer-Parallel Deep Neural Network Training
This paper investigates multilevel initialization strategies for training very deep neural networks with a layer-parallel multigrid solver. The scheme is based on the continuous interpretation of the training problem as …
Initialization and Regularization of Factorized Neural Layers
Factorized layers--operations parameterized by products of two or more matrices--occur in a variety of deep learning contexts, including compressed model training, certain types of knowledge distillation, and multi-head …
Knowledge DistillationModel CompressionTensor DecompositionUnsupervised Pre-trainingFractional moment-preserving initialization schemes for training deep neural networks
A traditional approach to initialization in deep neural networks (DNNs) is to sample the network weights randomly for preserving the variance of pre-activations. On the other hand, several studies show that during the tr…
Adaptive Signal Variances: CNN Initialization Through Modern Architectures
Deep convolutional neural networks (CNN) have achieved the unwavering confidence in its performance on image processing tasks. The CNN architecture constitutes a variety of different types of layers including the convolu…