TaskNorm: Rethinking Batch Normalization for Meta-Learning
Modern meta-learning approaches for image classification rely on increasingly deep networks to achieve state-of-the-art performance, making batch normalization an essential component of meta-learning pipelines. However, the hierarchical nature of the meta-learning setting presents several challenges that can render conventional batch normalization ineffective, giving rise to the need to rethink normalization in this setting. We evaluate a range of approaches to batch normalization for meta-learning scenarios, and develop a novel approach that we call TaskNorm. Experiments on fourteen datasets demonstrate that the choice of batch normalization has a dramatic effect on both classification accuracy and training time for both gradient based and gradient-free meta-learning approaches. Importantly, TaskNorm is found to consistently improve performance. Finally, we provide a set of best practices for normalization that will allow fair comparison of meta-learning algorithms.
Code (2)
Tasks
General Classificationimage-classificationImage ClassificationMeta-LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Rethinking Normalization and Elimination Singularity in Neural Networks
In this paper, we study normalization methods for neural networks from the perspective of elimination singularity. Elimination singularities correspond to the points on the training trajectory where neurons become consis…
image-classificationImage ClassificationInstance Segmentationobject-detection+3MetaNorm: Learning to Normalize Few-Shot Batches Across Domains
Batch normalization plays a crucial role when training deep neural networks. However, batch statistics become unstable with small batch sizes and are unreliable in the presence of distribution shifts. We propose MetaNorm…
Domain GeneralizationFew-Shot LearningMeta-LearningMeta Batch-Instance Normalization for Generalizable Person Re-Identification
Although supervised person re-identification (Re-ID) methods have shown impressive performance, they suffer from a poor generalization capability on unseen domains. Therefore, generalizable Re-ID has recently attracted g…
Data AugmentationDomain GeneralizationGeneralizable Person Re-identificationMeta-Learning+2Metadata Normalization
Batch Normalization (BN) and its variants have delivered tremendous success in combating the covariate shift induced by the training step of deep learning methods. While these techniques normalize feature distributions b…
Continual Normalization: Rethinking Batch Normalization for Online Continual Learning
Existing continual learning methods use Batch Normalization (BN) to facilitate training and improve generalization across tasks. However, the non-i.i.d and non-stationary nature of continual learning data, especially in …
Continual Learning