Kernel Modulation: A Parameter-Efficient Method for Training Convolutional Neural Networks
Deep Neural Networks, particularly Convolutional Neural Networks (ConvNets), have achieved incredible success in many vision tasks, but they usually require millions of parameters for good accuracy performance. With increasing applications that use ConvNets, updating hundreds of networks for multiple tasks on an embedded device can be costly in terms of memory, bandwidth, and energy. Approaches to reduce this cost include model compression and parameter-efficient models that adapt a subset of network layers for each new task. This work proposes a novel parameter-efficient kernel modulation (KM) method that adapts all parameters of a base network instead of a subset of layers. KM uses lightweight task-specialized kernel modulators that require only an additional 1.4% of the base network parameters. With multiple tasks, only the task-specialized KM weights are communicated and stored on the end-user device. We applied this method in training ConvNets for Transfer Learning and Meta-Learning scenarios. Our results show that KM delivers up to 9% higher accuracy than other parameter-efficient methods on the Transfer Learning benchmark.
Code (0)
등록된 구현이 없습니다.
Tasks
Meta-LearningModel CompressionTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
KernelDNA: Dynamic Kernel Sharing via Decoupled Naive Adapters
Dynamic convolution enhances model capacity by adaptively combining multiple kernels, yet faces critical trade-offs: prior works either (1) incur significant parameter overhead by scaling kernel numbers linearly, (2) com…
Computational Efficiencyimage-classificationImage ClassificationImage-specific Convolutional Kernel Modulation for Single Image Super-resolution
Recently, deep-learning-based super-resolution methods have achieved excellent performances, but mainly focus on training a single generalized deep network by feeding numerous samples. Yet intuitively, each image has its…
Image Super-ResolutionSuper-ResolutionFrequency Switching Mechanism for Parameter-E!cient Multi-Task Learning
Multi-task learning (MTL) aims to enable a single model to solve multiple tasks efficiently; however, current parameter-efficient fine-tuning (PEFT) methods remain largely limited to single-task adaptation. We introduce …
parameter-efficient fine-tuningMulti-Task LearningRep3D: Re-parameterize Large 3D Kernels with Low-Rank Receptive Modeling for Medical Imaging
In contrast to vision transformers, which model long-range dependencies through global self-attention, large kernel convolutions provide a more efficient and scalable alternative, particularly in high-resolution 3D volum…
Inductive BiasMedical Image AnalysisFlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
When designing Convolutional Neural Networks (CNNs), one must select the size\break of the convolutional kernels before training. Recent works show CNNs benefit from different kernel sizes at different layers, but explor…
DescriptiveImage ClassificationSequential Image ClassificationTime Series Analysis