LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networks
LOBSTER (LOss-Based SensiTivity rEgulaRization) is a method for training neural networks having a sparse topology. Let the sensitivity of a network parameter be the variation of the loss function with respect to the variation of the parameter. Parameters with low sensitivity, i.e. having little impact on the loss when perturbed, are shrunk and then pruned to sparsify the network. Our method allows to train a network from scratch, i.e. without preliminary learning or rewinding. Experiments on multiple architectures and datasets show competitive compression ratios with minimal computational overhead.
Code (0)
등록된 구현이 없습니다.
Tasks
SensitivitySimilar Papers 제목 키워드 기반
SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networks
Deep neural networks include millions of learnable parameters, making their deployment over resource-constrained devices problematic. SeReNe (Sensitivity-based Regularization of Neurons) is a method for learning sparse t…
SensitivityOn the Sensitivity of the Lasso to the Number of Predictor Variables
The Lasso is a computationally efficient regression regularization procedure that can produce sparse estimators when the number of predictors (p) is large. Oracle inequalities provide probability loss bounds for the Lass…
SensitivityLai Loss: A Novel Loss for Gradient Control
In the field of machine learning, traditional regularization methods tend to directly add regularization terms to the loss function. This paper introduces the "Lai loss", a novel loss design that integrates the regulariz…
SensitivityLearning Sparse Neural Networks via Sensitivity-Driven Regularization
The ever-increasing number of parameters in deep neural networks poses challenges for memory-limited applications. Regularize-and-prune methods aim at meeting these challenges by sparsifying the network weights. In this …
SensitivityAbsum: Simple Regularization Method for Reducing Structural Sensitivity of Convolutional Neural Networks
We propose Absum, which is a regularization method for improving adversarial robustness of convolutional neural networks (CNNs). Although CNNs can accurately recognize images, recent studies have shown that the convoluti…
Adversarial AttackAdversarial RobustnessSensitivity