$S^{2}$-LBI: Stochastic Split Linearized Bregman Iterations for Parsimonious Deep Learning
This paper proposes a novel Stochastic Split Linearized Bregman Iteration ($S^{2}$-LBI) algorithm to efficiently train the deep network. The $S^{2}$-LBI introduces an iterative regularization path with structural sparsity. Our $S^{2}$-LBI combines the computational efficiency of the LBI, and model selection consistency in learning the structural sparsity. The computed solution path intrinsically enables us to enlarge or simplify a network, which theoretically, is benefited from the dynamics property of our $S^{2}$-LBI algorithm. The experimental results validate our $S^{2}$-LBI on MNIST and CIFAR-10 dataset. For example, in MNIST, we can either boost a network with only 1.5K parameters (1 convolutional layer of 5 filters, and 1 FC layer), achieves 98.40\% recognition accuracy; or we simplify $82.5\%$ of parameters in LeNet-5 network, and still achieves the 98.47\% recognition accuracy. In addition, we also have the learning results on ImageNet, which will be added in the next version of our report.
Code (0)
등록된 구현이 없습니다.
Tasks
Computational EfficiencyModel SelectionSimilar Papers 제목 키워드 기반
The Linearized Bregman Method via Split Feasibility Problems: Analysis and Generalizations
The linearized Bregman method is a method to calculate sparse solutions to systems of linear equations. We formulate this problem as a split feasibility problem, propose an algorithmic framework based on Bregman projecti…
Linearized Bregman Iterations for Sparse Spiking Neural Networks
Spiking Neural Networks (SNNs) offer an energy efficient alternative to conventional Artificial Neural Networks (ANNs) but typically still require a large number of parameters. This work introduces Linearized Bregman Ite…
Linearized Bregman Iterations for Automatic Optical Fiber Fault Analysis
Supervision of the physical layer of optical networks is an extremely relevant subject. To detect fiber faults, single-ended solutions such as the Optical Time Domain Reflectometer (OTDR) allow for precise measurements o…
parameter estimationSpecificityFrom Social to Individuals: a Parsimonious Path of Multi-level Models for Crowdsourced Preference Aggregation
In crowdsourced preference aggregation, it is often assumed that all the annotators are subject to a common preference or social utility function which generates their comparison behaviors in experiments. However, in rea…
Sparse Training of Neural Networks based on Multilevel Mirror Descent
We introduce a dynamic sparse training algorithm based on linearized Bregman iterations / mirror descent that exploits the naturally incurred sparsity by alternating between periods of static and dynamic sparsity pattern…