Kernel Normalized Convolutional Networks
Existing convolutional neural network architectures frequently rely upon batch normalization (BatchNorm) to effectively train the model. BatchNorm, however, performs poorly with small batch sizes, and is inapplicable to differential privacy. To address these limitations, we propose the kernel normalization (KernelNorm) and kernel normalized convolutional layers, and incorporate them into kernel normalized convolutional networks (KNConvNets) as the main building blocks. We implement KNConvNets corresponding to the state-of-the-art ResNets while forgoing the BatchNorm layers. Through extensive experiments, we illustrate that KNConvNets achieve higher or competitive performance compared to the BatchNorm counterparts in image classification and semantic segmentation. They also significantly outperform their batch-independent competitors including those based on layer and group normalization in non-private and differentially private training. Given that, KernelNorm combines the batch-independence property of layer and group normalization with the performance advantage of BatchNorm.
Code (1)
Tasks
Federated Learningimage-classificationImage ClassificationSemantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Normalized Convolutional Neural Network
In this paper, we propose Normalized Convolutional Neural Network(NCNN). NCNN is more fitted to a convolutional operator than other nomralizaiton methods. The normalized process is similar to a normalization methods, but…
On the Spectral Bias of Convolutional Neural Tangent and Gaussian Process Kernels
We study the properties of various over-parametrized convolutional neural architectures through their respective Gaussian process and neural tangent kernels. We prove that, with normalized multi-channel input and ReLU ac…
Kernel Normalized Convolutional Networks for Privacy-Preserving Machine Learning
Normalization is an important but understudied challenge in privacy-related application domains such as federated learning (FL), differential privacy (DP), and differentially private federated learning (DP-FL). While the…
Federated LearningPrivacy PreservingNormalized Cut Loss for Weakly-supervised CNN Segmentation
Most recent semantic segmentation methods train deep convolutional neural networks with fully annotated masks requiring pixel-accuracy for good quality training. Common weakly-supervised approaches generate full masks fr…
Interactive SegmentationSegmentationSemantic SegmentationObject Tracking in Hyperspectral Videos with Convolutional Features and Kernelized Correlation Filter
Target tracking in hyperspectral videos is a new research topic. In this paper, a novel method based on convolutional network and Kernelized Correlation Filter (KCF) framework is presented for tracking objects of interes…
Object Tracking