Weight Normalization based Quantization for Deep Neural Network Compression

2019-07-01 · Wen-Pu Cai, Wu-Jun Li

With the development of deep neural networks, the size of network models becomes larger and larger. Model compression has become an urgent need for deploying these network models to mobile or embedded devices. Model quantization is a representative model compression technique. Although a lot of quantization methods have been proposed, many of them suffer from a high quantization error caused by a long-tail distribution of network weights. In this paper, we propose a novel quantization method, called weight normalization based quantization (WNQ), for model compression. WNQ adopts weight normalization to avoid the long-tail distribution of network weights and subsequently reduces the quantization error. Experiments on CIFAR-100 and ImageNet show that WNQ can outperform other baselines to achieve state-of-the-art performance.

📄 PDF Abstract BibTeX arXiv:1907.00593

Code (0)

등록된 구현이 없습니다.

Tasks

Model Compression Neural Network Compression Quantization

Methods 이 논문이 사용한 방법론

Weight Normalization Weight Normalization is a normalization method for training neural networks. It is inspired by batch normalization,…

Pruning Ternary Quantization

2021-07-23 · Dan Liu, Xi Chen, Jie Fu, Chen Ma 외

Inference time, model size, and accuracy are three key factors in deep model compression. Most of the existing work addresses these three key factors separately as it is difficult to optimize them all at the same time. F…

image-classification Image Classification Model Compression object-detection+2

N-Ary Quantization for CNN Model Compression and Inference Acceleration

2019-05-01 · ICLR 2019 5 · Günther Schindler, Wolfgang Roth, Franz Pernkopf, Holger Fröning

The tremendous memory and computational complexity of Convolutional Neural Networks (CNNs) prevents the inference deployment on resource-constrained systems. As a result, recent research focused on CNN optimization techn…

Clustering Model Compression Quantization

Magic for the Age of Quantized DNNs

2024-03-22 · Yoshihide Sawada, Ryuji Saiin, Kazuma Suetake

Recently, the number of parameters in DNNs has explosively increased, as exemplified by LLMs (Large Language Models), making inference on small-scale computers more difficult. Model compression technology is, therefore, …

Model Compression Quantization

PolarQuant: Optimal Gaussian Weight Quantization via Hadamard Rotation for LLM Compression

2026-03-30 · Caio Vicentino arxiv

We present PolarQuant, a post-training weight quantization method for large language models (LLMs) that exploits the distributional structure of neural network weights to achieve near-lossless compression. PolarQuant ope…

Norm Tweaking: High-performance Low-bit Quantization of Large Language Models

2023-09-06 · Liang Li, Qingyuan Li, Bo Zhang, Xiangxiang Chu

As the size of large language models (LLMs) continues to grow, model compression without sacrificing accuracy has become a crucial challenge for deployment. While some quantization methods, such as GPTQ, have made progre…