Is Feature Diversity Necessary in Neural Network Initialization?
Standard practice in training neural networks involves initializing the weights in an independent fashion. The results of recent work suggest that feature "diversity" at initialization plays an important role in training the network. However, other initialization schemes with reduced feature diversity have also been shown to be viable. In this work, we conduct a series of experiments aimed at elucidating the importance of feature diversity at initialization. We show that a complete lack of diversity is harmful to training, but its effects can be counteracted by a relatively small addition of noise - even the noise in standard non-deterministic GPU computations is sufficient. Furthermore, we construct a deep convolutional network with identical features at initialization and almost all of the weights initialized at 0 that can be trained to reach accuracy matching its standard-initialized counterpart.
Code (1)
Tasks
DiversityGPUSimilar Papers 제목 키워드 기반
Beyond Signal Propagation: Is Feature Diversity Necessary in Deep Neural Network Initialization?
Deep neural networks are typically initialized with random weights, with variances chosen to facilitate signal propagation and stable gradients. It is also believed that diversity of features is an important property of …
DiversityGPUA Survey on Activation Functions and their relation with Xavier and He Normal Initialization
In artificial neural network, the activation function and the weight initialization method play important roles in training and performance of a neural network. The question arises is what properties of a function are im…
RelationSurveyInitialization is Half the Battle: Generating Diverse Images from a Guidance Potential Posterior
Despite the remarkable fidelity of generative models, they frequently suffer from mode collapse. Existing strategies for enhancing diversity predominantly focus on intervening during the generation trajectory. We identif…
Self-supervised Assisted Active Learning for Skin Lesion Segmentation
Label scarcity has been a long-standing issue for biomedical image segmentation, due to high annotation costs and professional requirements. Recently, active learning (AL) strategies strive to reduce annotation costs by …
Active LearningDiversityImage SegmentationLesion Segmentation+4Enhancing Diversity in Multi-objective Feature Selection
Feature selection plays a pivotal role in the data preprocessing and model-building pipeline, significantly enhancing model performance, interpretability, and resource efficiency across diverse domains. In population-bas…
Diversityfeature selection