Differentiable Product Quantization for End-to-End Embedding Compression
Embedding layers are commonly used to map discrete symbols into continuous embedding vectors that reflect their semantic meanings. Despite their effectiveness, the number of parameters in an embedding layer increases linearly with the number of symbols and poses a critical challenge on memory and storage constraints. In this work, we propose a generic and end-to-end learnable compression framework termed differentiable product quantization (DPQ). We present two instantiations of DPQ that leverage different approximation techniques to enable differentiability in end-to-end learning. Our method can readily serve as a drop-in alternative for any existing embedding layer. Empirically, DPQ offers significant compression ratios (14-238$\times$) at negligible or no performance cost on 10 datasets across three different language tasks.
Code (2)
Tasks
QuantizationSimilar Papers 제목 키워드 기반
Differentiable Product Quantization for Learning Compact Embedding Layers
Embedding layers are commonly used to map discrete symbols into continuous embedding vectors that reflect their semantic meanings. Despite their effectiveness, the number of parameters in an embedding layer increases lin…
QuantizationLearning Compact Embedding Layers via Differentiable Product Quantization
Embedding layers are commonly used to map discrete symbols into continuous embedding vectors that reflect their semantic meanings. Despite their effectiveness, the number of parameters in an embedding layer increases lin…
QuantizationDifferentiable Product Quantization for Memory Efficient Camera Relocalization
Camera relocalization relies on 3D models of the scene with a large memory footprint that is incompatible with the memory budget of several applications. One solution to reduce the scene memory size is map compression by…
Camera RelocalizationQuantizationJointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance
Recently, Information Retrieval community has witnessed fast-paced advances in Dense Retrieval (DR), which performs first-stage retrieval with embedding-based search. Despite the impressive ranking performance, previous …
CPUGPUInformation RetrievalQuantization+1Universally Quantized Neural Compression
A popular approach to learning encoders for lossy compression is to use additive uniform noise during training as a differentiable approximation to test-time quantization. We demonstrate that a uniform noise channel can …
Quantization