Weight Squeezing: Reparameterization for Compression and Fast Inference
In this work, we present a novel approach for simultaneous knowledge transfer and model compression called Weight Squeezing. With this method, we perform knowledge transfer from a pre-trained teacher model by learning the mapping from its weights to smaller student model weights, without significant loss of model accuracy. We applied Weight Squeezing combined with Knowledge Distillation to a pre-trained text classification model, and compared it to various knowledge transfer and model compression methods on several downstream text classification tasks. We observed that our approach produces better results than Knowledge Distillation methods without any loss in inference speed. We also compared Weight Squeezing with Low Rank Factorization methods and observed that our method is significantly faster at inference while being competitive in terms of accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationModel Compressiontext-classificationText ClassificationTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Weight Squeezing: Reparameterization for Knowledge Transfer and Model Compression
In this work, we present a novel approach to simultaneous knowledge transfer and model compression called \textbf{Weight Squeezing}. With this method, we perform knowledge transfer from a teacher model \textbf{by learnin…
Model Compressiontext-classificationText ClassificationTransfer LearningWeight Squeezing: Reparameterization for Knowledge Transfer and Model Compression
In this work, we present a novel approach for simultaneous knowledge transfer and model compression called Weight Squeezing. With this method, we perform knowledge transfer from a teacher model by learning the mapping fr…
General ClassificationModel Compressiontext-classificationText Classification+1Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method
Deploying large language models (LLMs) is challenging due to their massive parameters and high computational costs. Ultra low-bit quantization can significantly reduce storage and accelerate inference, but extreme compre…
Scalable Model Compression by Entropy Penalized Reparameterization
We describe a simple and general neural network weight compression approach, in which the network parameters (weights and biases) are represented in a "latent" space, amounting to a reparameterization. This space is equi…
General ClassificationmodelModel CompressionRepControlNet: ControlNet Reparameterization
With the wide application of diffusion model, the high cost of inference resources has became an important bottleneck for its universal application. Controllable generation, such as ControlNet, is one of the key research…
Model Compression