Compressing Deep Convolutional Networks using Vector Quantization

2014-12-18 · Yunchao Gong, Liu Liu, Ming Yang, Lubomir Bourdev

Deep convolutional neural networks (CNN) has become the most promising method for object recognition, repeatedly demonstrating record breaking results for image classification and object detection in recent years. However, a very deep CNN generally involves many layers with millions of parameters, making the storage of the network model to be extremely large. This prohibits the usage of deep CNNs on resource limited hardware, especially cell phones or other embedded devices. In this paper, we tackle this model storage issue by investigating information theoretical vector quantization methods for compressing the parameters of CNNs. In particular, we have found in terms of compressing the most storage demanding dense connected layers, vector quantization methods have a clear gain over existing matrix factorization methods. Simply applying k-means clustering to the weights or conducting product quantization can lead to a very good balance between model size and recognition accuracy. For the 1000-category classification task in the ImageNet challenge, we are able to achieve 16-24 times compression of the network with only 1% loss of classification accuracy using the state-of-the-art CNN.

📄 PDF Abstract BibTeX arXiv:1412.6115

Code (0)

등록된 구현이 없습니다.

Tasks

ClassificationClusteringGeneral Classification image-classification Image Classification object-detection Object Detection Object Recognition Quantization

Methods 이 논문이 사용한 방법론

k-Means Clustering k-Means Clustering is a clustering algorithm that divides a training set into $k$ different clusters of examples that are near each other. It works by initializing $k$…

CCQ: Convolutional Code for Extreme Low-bit Quantization in LLMs

2025-07-09 · Zhaojing Zhou, Xunchao Li, Minghao Li, Handi Zhang 외 arxiv

The rapid scaling of Large Language Models (LLMs) elevates inference costs and compounds substantial deployment barriers. While quantization to 8 or 4 bits mitigates this, sub-3-bit methods face severe accuracy, scalabil…

And the Bit Goes Down: Revisiting the Quantization of Neural Networks

2019-07-12 · ICLR 2020 1 · Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham 외

In this paper, we address the problem of reducing the memory footprint of convolutional network architectures. We introduce a vector quantization method that aims at preserving the quality of the reconstruction of the ne…

CPU Quantization

Optimal and Near-Optimal Adaptive Vector Quantization

2024-02-05 · Ran Ben-Basat, Yaniv Ben-Itzhak, Michael Mitzenmacher, Shay Vargaftik

Quantization is a fundamental optimization for many machine-learning use cases, including compressing gradients, model weights and activations, and datasets. The most accurate form of quantization is \emph{adaptive}, whe…

Quantization

RQ-MoE: Residual Quantization via Mixture of Experts for Efficient Input-Dependent Vector Compression

2026-05-14 · Zhengjia Zhong, Shuyan Ke, Zaizhou Lin, Jiaqi Song 외 arxiv

Vector quantization is a fundamental tool for compressing high-dimensional embeddings, yet existing multi-codebook methods rely on static codebooks that limit expressiveness under heterogeneous data geometry. While recen…

Pyramid Vector Quantization for Deep Learning

2017-04-10 · Vincenzo Liguori

This paper explores the use of Pyramid Vector Quantization (PVQ) to reduce the computational cost for a variety of neural networks (NNs) while, at the same time, compressing the weights that describe them. This is based …

Deep Learning Quantization