Mode Normalization
Normalization methods are a central building block in the deep learning toolbox. They accelerate and stabilize training, while decreasing the dependence on manually tuned learning rate schedules. When learning from multi-modal distributions, the effectiveness of batch normalization (BN), arguably the most prominent normalization method, is reduced. As a remedy, we propose a more flexible approach: by extending the normalization to more than a single mean and variance, we detect modes of data on-the-fly, jointly normalizing samples that share common features. We demonstrate that our method outperforms BN and other widely used normalization techniques in several experiments, including single and multi-task datasets.
Code (2)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
We investigate the reasons for the performance degradation incurred with batch-independent normalization. We find that the prototypical techniques of layer normalization and instance normalization both induce the appeara…
Extended Batch Normalization
Batch normalization (BN) has become a standard technique for training the modern deep networks. However, its effectiveness diminishes when the batch size becomes smaller, since the batch statistics estimation becomes ina…
Image ClassificationLearning Graph Normalization for Graph Neural Networks
Graph Neural Networks (GNNs) have attracted considerable attention and have emerged as a new promising paradigm to process graph-structured data. GNNs are usually stacked to multiple layers and the node representations i…
Graph ClassificationGraph RegressionLink PredictionNode ClassificationAFN: Adaptive Fusion Normalization via an Encoder-Decoder Framework
The success of deep learning is inseparable from normalization layers. Researchers have proposed various normalization functions, and each of them has both advantages and disadvantages. In response, efforts have been mad…
DecoderDomain Generalizationimage-classificationImage ClassificationNew Interpretations of Normalization Methods in Deep Learning
In recent years, a variety of normalization methods have been proposed to help train neural networks, such as batch normalization (BN), layer normalization (LN), weight normalization (WN), group normalization (GN), etc. …
Deep LearningLEMMA