On Lipschitz Regularization of Convolutional Layers using Toeplitz Matrix Theory
This paper tackles the problem of Lipschitz regularization of Convolutional Neural Networks. Lipschitz regularity is now established as a key property of modern deep learning with implications in training stability, generalization, robustness against adversarial examples, etc. However, computing the exact value of the Lipschitz constant of a neural network is known to be NP-hard. Recent attempts from the literature introduce upper bounds to approximate this constant that are either efficient but loose or accurate but computationally expensive. In this work, by leveraging the theory of Toeplitz matrices, we introduce a new upper bound for convolutional layers that is both tight and easy to compute. Based on this result we devise an algorithm to train Lipschitz regularized Convolutional Neural Networks.
Code (2)
Similar Papers 제목 키워드 기반
Lipschitz Bound Analysis of Neural Networks
Lipschitz Bound Estimation is an effective method of regularizing deep neural networks to make them robust against adversarial attacks. This is useful in a variety of applications ranging from reinforcement learning to a…
Rolling Shutter CorrectionEfficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iteration
Since the control of the Lipschitz constant has a great impact on the training stability, generalization, and robustness of neural networks, the estimation of this value is nowadays a real scientific challenge. In this p…
Ortho-Shot: Low Displacement Rank Regularization with Data Augmentation for Few-Shot Learning
In few-shot classification, the primary goal is to learn representations from a few samples that generalize well for novel classes. In this paper, we propose an efficient low displacement rank (LDR) regularization strate…
Data AugmentationDiversityFew-Shot LearningAsymptotic Singular Value Distribution of Linear Convolutional Layers
In convolutional neural networks, the linear transformation of multi-channel two-dimensional convolutional layers with linear convolution is a block matrix with doubly Toeplitz blocks. Although a "wrapping around" operat…
Lipschitz-bounded 1D convolutional neural networks using the Cayley transform and the controllability Gramian
We establish a layer-wise parameterization for 1D convolutional neural networks (CNNs) with built-in end-to-end robustness guarantees. In doing so, we use the Lipschitz constant of the input-output mapping characterized …