SHADE: SHAnnon DEcay Information-Based Regularization for Deep Learning
Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay. The originality of the approach is to define a prior based on conditional entropy, which explicitly decouples the learning of invariant representations in the regularizer and the learning of correlations between inputs and labels in the data fitting term. We explain why this quantity makes our model able to achieve invariance with respect to input variations. We empirically validate the efficiency of our approach to improve classification performances compared to standard regularization schemes on several standard architectures.
Code (0)
등록된 구현이 없습니다.
Tasks
Deep LearningGeneral ClassificationSimilar Papers 제목 키워드 기반
SHADE: Information Based Regularization for Deep Learning
Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay. The originality of the approach is to define …
Deep LearningGeneral ClassificationSHADE: Information-Based Regularization for Deep Learning
Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay. The originality of the approach is to define …
Deep LearningGeneral ClassificationImproving Generalization by Controlling Label-Noise Information in Neural Network Weights
In the presence of noisy or incorrect labels, neural networks have the undesirable tendency to memorize information about the noise. Standard regularization techniques such as dropout, weight decay or data augmentation s…
Data AugmentationGeneralization BoundsLearning with noisy labelsMemorizationOUIDecay: Adaptive Layer-wise Weight Decay for CNNs Using Online Activation Patterns
Weight decay remains one of the most widely used regularization mechanisms for training convolutional neural networks, yet it is still commonly applied as a fixed coefficient shared by all layers throughout training. Thi…
Stable Weight Decay Regularization
Weight decay is a popular regularization technique for training of deep neural networks. Modern deep learning libraries mainly use $L_{2}$ regularization as the default implementation of weight decay. \citet{loshchilov20…