Variational Network Quantization
In this paper, the preparation of a neural network for pruning and few-bit quantization is formulated as a variational inference problem. To this end, a quantizing prior that leads to a multi-modal, sparse posterior distribution over weights, is introduced and a differentiable Kullback-Leibler divergence approximation for this prior is derived. After training with Variational Network Quantization, weights can be replaced by deterministic quantization values with small to negligible loss of task accuracy (including pruning by setting weights to 0). The method does not require fine-tuning after quantization. Results are shown for ternary quantization on LeNet-5 (MNIST) and DenseNet (CIFAR-10).
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationVariational InferenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization
One noted issue of vector-quantized variational autoencoder (VQ-VAE) is that the learned discrete representation uses only a fraction of the full capacity of the codebook, also known as codebook collapse. We hypothesize …
QuantizationOn the relation between trainability and dequantization of variational quantum learning models
The quest for successful variational quantum machine learning (QML) relies on the design of suitable parametrized quantum circuits (PQCs), as analogues to neural networks in classical machine learning. Successful QML mod…
Quantum Machine LearningRelationSoft then Hard: Rethinking the Quantization in Neural Image Compression
Quantization is one of the core components in lossy image compression. For neural image compression, end-to-end optimization requires differentiable approximations of quantization, which can generally be grouped into thr…
Image CompressionQuantizationUncertainty-Preserving QBNNs: Multi-Level Quantization of SVI-Based Bayesian Neural Networks for Image Classification
Bayesian Neural Networks (BNNs) provide principled uncertainty quantification but suffer from substantial computational and memory overhead compared to deterministic networks. While quantization techniques have successfu…
Image ClassificationVariational Bayesian Quantization
We propose a novel algorithm for quantizing continuous latent representations in trained models. Our approach applies to deep probabilistic models, such as variational autoencoders (VAEs), and enables both data and model…
Image CompressionModel CompressionQuantizationWord Embeddings