Embedding Compression with Isotropic Iterative Quantization
Continuous representation of words is a standard component in deep learning-based NLP models. However, representing a large vocabulary requires significant memory, which can cause problems, particularly on resource-constrained platforms. Therefore, in this paper we propose an isotropic iterative quantization (IIQ) approach for compressing embedding vectors into binary ones, leveraging the iterative quantization technique well established for image retrieval, while satisfying the desired isotropic property of PMI based models. Experiments with pre-trained embeddings (i.e., GloVe and HDC) demonstrate a more than thirty-fold compression ratio with comparable and sometimes even improved performance over the original real-valued embedding vectors.
Code (0)
등록된 구현이 없습니다.
Tasks
Image RetrievalQuantizationRetrievalMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Physics-Informed Vector Quantized Autoencoder for Data Compression of Turbulent Flow
Analyzing large-scale data from simulations of turbulent flows is memory intensive, requiring significant resources. This major challenge highlights the need for data compression techniques. In this study, we apply a phy…
Data CompressionQuantizationRetraining-Based Iterative Weight Quantization for Deep Neural Networks
Model compression has gained a lot of attention due to its ability to reduce hardware resource requirements significantly while maintaining accuracy of DNNs. Model compression is especially useful for memory-intensive re…
Model CompressionQuantizationEmbedding Compression for Efficient Re-Identification
Real world re-identfication (ReID) algorithms aim to map new observations of an object to previously recorded instances. These systems are often constrained by quantity and size of the stored embeddings. To combat this s…
Dimensionality ReductionQuantizationWhen Is 0.1% Enough? Analyzing the Combined Effects of Dimensionality Reduction and Quantization on Text Embedding Compression
Recent high-performing text embedding models often output high-dimensional real-valued vectors, resulting in substantial storage and computational costs. To address this issue, compression methods based on dimensionality…
Dimensionality ReductionOPQ: Compressing Deep Neural Networks with One-shot Pruning-Quantization
As Deep Neural Networks (DNNs) usually are overparameterized and have millions of weight parameters, it is challenging to deploy these large DNN models on resource-constrained hardware platforms, e.g., smartphones. Numer…
Quantization