Lipschitz-bounded 1D convolutional neural networks using the Cayley transform and the controllability Gramian
We establish a layer-wise parameterization for 1D convolutional neural networks (CNNs) with built-in end-to-end robustness guarantees. In doing so, we use the Lipschitz constant of the input-output mapping characterized by a CNN as a robustness measure. We base our parameterization on the Cayley transform that parameterizes orthogonal matrices and the controllability Gramian of the state space representation of the convolutional layers. The proposed parameterization by design fulfills linear matrix inequalities that are sufficient for Lipschitz continuity of the CNN, which further enables unconstrained training of Lipschitz-bounded 1D CNNs. Finally, we train Lipschitz-bounded 1D CNNs for the classification of heart arrythmia data and show their improved robustness.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Orthogonalizing Convolutional Layers with the Cayley Transform
Recent work has highlighted several advantages of enforcing orthogonality in the weight layers of deep networks, such as maintaining the stability of activations, preserving gradient norms, and enhancing adversarial robu…
Adversarial RobustnessEfficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform
Strictly enforcing orthonormality constraints on parameter matrices has been shown advantageous in deep learning. This amounts to Riemannian optimization on the Stiefel manifold, which, however, is computationally expens…
Riemannian optimizationThe Lipschitz Constant of Self-Attention
Lipschitz constants of neural networks have been explored in various contexts in deep learning, such as provable adversarial robustness, estimating Wasserstein distance, stabilising training of GANs, and formulating inve…
Adversarial RobustnessLanguage ModellingUniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widths
We consider deep neural networks with a Lipschitz continuous activation function and with weight matrices of variable widths. We establish a uniform convergence analysis framework in which sufficient conditions on weight…
LipKernel: Lipschitz-Bounded Convolutional Neural Networks via Dissipative Layers
We propose a novel layer-wise parameterization for convolutional neural networks (CNNs) that includes built-in robustness guarantees by enforcing a prescribed Lipschitz bound. Each layer in our parameterization is design…
Autonomous Vehicles