Revisiting Parameter Sharing for Automatic Neural Channel Number Search
Recent advances in neural architecture search inspire many channel number search algorithms~(CNS) for convolutional neural networks. To improve searching efficiency, parameter sharing is widely applied, which reuses parameters among different channel configurations. Nevertheless, it is unclear how parameter sharing affects the searching process. In this paper, we aim at providing a better understanding and exploitation of parameter sharing for CNS. Specifically, we propose affine parameter sharing~(APS) as a general formulation to unify and quantitatively analyze existing channel search algorithms. It is found that with parameter sharing, weight updates of one architecture can simultaneously benefit other candidates. However, it also results in less confidence in choosing good architectures. We thus propose a new strategy of parameter sharing towards a better balance between training efficiency and architecture discrimination. Extensive analysis and experiments demonstrate the superiority of the proposed strategy in channel configuration against many state-of-the-art counterparts on benchmark datasets.
Code (1)
Tasks
Neural Architecture SearchSimilar Papers 제목 키워드 기반
Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing
Sharing parameters in multi-agent deep reinforcement learning has played an essential role in allowing algorithms to scale to a large number of agents. Parameter sharing between agents significantly decreases the number …
Deep Reinforcement LearningMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+1Drastically Reducing the Number of Trainable Parameters in Deep CNNs by Inter-layer Kernel-sharing
Deep convolutional neural networks (DCNNs) have become the state-of-the-art (SOTA) approach for many computer vision tasks: image classification, object detection, semantic segmentation, etc. However, most SOTA networks …
Edge-computingimage-classificationImage Classificationobject-detection+2Selectively Sharing Experiences Improves Multi-Agent Reinforcement Learning
We present a novel multi-agent RL approach, Selective Multi-Agent Prioritized Experience Relay, in which agents share with other agents a limited number of transitions they observe during training. The intuition behind t…
Multi-agent Reinforcement Learningreinforcement-learningReinforcement LearningRevisiting Dynamic Convolution via Matrix Decomposition
Recent research in dynamic convolution shows substantial performance boost for efficient CNNs, due to the adaptive aggregation of K static convolution kernels. It has two limitations: (a) it increases the number of convo…
Dimensionality ReductionRecNets: Channel-wise Recurrent Convolutional Neural Networks
In this paper, we introduce Channel-wise recurrent convolutional neural networks (RecNets), a family of novel, compact neural network architectures for computer vision tasks inspired by recurrent neural networks (RNNs). …
General Classificationimage-classificationImage Classification