Soft Weight-Sharing for Neural Network Compression
The success of deep learning in numerous application domains created the de- sire to run and train them on mobile devices. This however, conflicts with their computationally, memory and energy intense nature, leading to a growing interest in compression. Recent work by Han et al. (2015a) propose a pipeline that involves retraining, pruning and quantization of neural network weights, obtaining state-of-the-art compression rates. In this paper, we show that competitive compression rates can be achieved by using a version of soft weight-sharing (Nowlan & Hinton, 1992). Our method achieves both quantization and pruning in one simple (re-)training procedure. This point of view also exposes the relation between compression and the minimum description length (MDL) principle.
Code (3)
Tasks
Neural Network CompressionQuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Improved Bayesian Compression
Compression of Neural Networks (NN) has become a highly studied topic in recent years. The main reason for this is the demand for industrial scale usage of NNs such as deploying them on mobile devices, storing them effic…
Model CompressionBit-serial Weight Pools: Compression and Arbitrary Precision Execution of Neural Networks on Resource Constrained Processors
Applications of neural networks on edge systems have proliferated in recent years but the ever-increasing model size makes neural networks not able to deploy on resource-constrained microcontrollers efficiently. We propo…
Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression
Large Language Models (LLMs) have achieved remarkable breakthroughs. However, the huge number of parameters in LLMs require significant amount of memory storage in inference, which prevents their practical deployment in …
Language ModelingLanguage ModellingLarge Language ModelModel CompressionCross Domain Model Compression by Structurally Weight Sharing
Regular model compression methods focus on RGB input. While cross domain tasks demand more DNN models, each domain often needs its own model. Consequently, for such tasks, the storage cost, memory footprint and computati…
Action RecognitionGraph EmbeddingmodelModel Compression+2Exploiting Domain Knowledge via Grouped Weight Sharing with Application to Text Categorization
A fundamental advantage of neural models for NLP is their ability to learn representations from scratch. However, in practice this often means ignoring existing external linguistic resources, e.g., WordNet or domain spec…
General ClassificationModel CompressionText Categorization