The Singular Values of Convolutional Layers
We characterize the singular values of the linear transformation associated with a standard 2D multi-channel convolutional layer, enabling their efficient computation. This characterization also leads to an algorithm for projecting a convolutional layer onto an operator-norm ball. We show that this is an effective regularizer; for example, it improves the test error of a deep residual network using batch normalization on CIFAR-10 from 6.2\% to 5.3\%.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Towards Practical Control of Singular Values of Convolutional Layers
In general, convolutional neural networks (CNNs) are easy to train, but their essential properties, such as generalization error and adversarial robustness, are hard to control. Recent research demonstrated that singular…
Adversarial RobustnessA Frobenius norm regularization method for convolutional kernels to avoid unstable gradient problem
Convolutional neural network is a very important model of deep learning. It can help avoid the exploding/vanishing gradient problem and improve the generalizability of a neural network if the singular values of the Jacob…
Asymptotic Singular Value Distribution of Linear Convolutional Layers
In convolutional neural networks, the linear transformation of multi-channel two-dimensional convolutional layers with linear convolution is a block matrix with doubly Toeplitz blocks. Although a "wrapping around" operat…
ManifoldFlow: SPD-Relaxed Stiefel Layers with Learnable Singular Spectrum
Orthogonal and Stiefel layers give neural weights exact spectral control, but they also impose a strong modeling constraint: all represented singular values are fixed at one. Many settings that benefit from an orthonorma…
The SVD of Convolutional Weights: A CNN Interpretability Framework
Deep neural networks used for image classification often use convolutional filters to extract distinguishing features before passing them to a linear classifier. Most interpretability literature focuses on providing sema…
image-classificationImage Classification