Channel redundancy and overlap in convolutional neural networks with channel-wise NNK graphs
Feature spaces in the deep layers of convolutional neural networks (CNNs) are often very high-dimensional and difficult to interpret. However, convolutional layers consist of multiple channels that are activated by different types of inputs, which suggests that more insights may be gained by studying the channels and how they relate to each other. In this paper, we first analyze theoretically channel-wise non-negative kernel (CW-NNK) regression graphs, which allow us to quantify the overlap between channels and, indirectly, the intrinsic dimension of the data representation manifold. We find that redundancy between channels is significant and varies with the layer depth and the level of regularization during training. Additionally, we observe that there is a correlation between channel overlap in the last convolutional layer and generalization performance. Our experimental results demonstrate that these techniques can lead to a better understanding of deep representations.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Channel Compression: Rethinking Information Redundancy among Channels in CNN Architecture
Model compression and acceleration are attracting increasing attentions due to the demand for embedded devices and mobile applications. Research on efficient convolutional neural networks (CNNs) aims at removing feature …
Acoustic Scene ClassificationEvent Detectionimage-classificationImage Classification+3Multi-Dimensional Pruning: A Unified Framework for Model Compression
In this work, we propose a unified model compression framework called Multi-Dimensional Pruning (MDP) to simultaneously compress the convolutional neural networks (CNNs) on multiple dimensions. In contrast to the existin…
Model CompressionLST-Net: Learning a Convolutional Neural Network with a Learnable Sparse Transform
The 2D convolutional (Conv2d) layer is the fundamental element to a deep convolutional neural network (CNN). Despite the great success of CNN, the conventional Conv2d is still limited in effectively reducing the spatial …
Exploiting Channel Similarity for Accelerating Deep Convolutional Neural Networks
To address the limitations of existing magnitude-based pruning algorithms in cases where model weights or activations are of large and similar magnitude, we propose a novel perspective to discover parameter redundancy am…
ClusteringMultichannel Sound Event Detection Using 3D Convolutional Neural Networks for Learning Inter-channel Features
In this paper, we propose a stacked convolutional and recurrent neural network (CRNN) with a 3D convolutional neural network (CNN) in the first layer for the multichannel sound event detection (SED) task. The 3D CNN enab…
Event DetectionSound Event Detection