A Framework for Neural Network Pruning Using Gibbs Distributions
Modern deep neural networks are often too large to use in many practical scenarios. Neural network pruning is an important technique for reducing the size of such models and accelerating inference. Gibbs pruning is a novel framework for expressing and designing neural network pruning methods. Combining approaches from statistical physics and stochastic regularization methods, it can train and prune a network simultaneously in such a way that the learned weights and pruning mask are well-adapted for each other. It can be used for structured or unstructured pruning and we propose a number of specific methods for each. We compare our proposed methods to a number of contemporary neural network pruning methods and find that Gibbs pruning outperforms them. In particular, we achieve a new state-of-the-art result for pruning ResNet-56 with the CIFAR-10 dataset.
Code (1)
Tasks
Network PruningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A mixing time bound for Gibbs sampling from log-smooth log-concave distributions
The Gibbs sampler, also known as the coordinate hit-and-run algorithm, is a Markov chain that is widely used to draw samples from probability distributions in arbitrary dimensions. At each iteration of the algorithm, a r…
Leveraging PAC-Bayes Theory and Gibbs Distributions for Generalization Bounds with Complexity Measures
In statistical learning theory, a generalization bound usually involves a complexity measure imposed by the considered theoretical framework. This limits the scope of such bounds, as other forms of capacity measures or r…
Generalization BoundsLearning TheoryReconsidering Dependency Networks from an Information Geometry Perspective
Dependency networks (Heckerman et al., 2000) provide a flexible framework for modeling complex systems with many variables by combining independently learned local conditional distributions through pseudo-Gibbs sampling.…
Diffusive Gibbs Sampling
The inadequate mixing of conventional Markov Chain Monte Carlo (MCMC) methods for multi-modal distributions presents a significant challenge in practical applications such as Bayesian inference and molecular dynamics. Ad…
Bayesian InferenceDenoisingNon-asymptotic analysis of Langevin-type Monte Carlo algorithms
We study Langevin-type algorithms for sampling from Gibbs distributions such that the potentials are dissipative and their weak gradients have finite moduli of continuity not necessarily convergent to zero. Our main resu…
Vocal Bursts Type Prediction