Aggregated Learning: A Deep Learning Framework Based on Information-Bottleneck Vector Quantization
Based on the notion of information bottleneck (IB), we formulate a quantization problem called "IB quantization". We show that IB quantization is equivalent to learning based on the IB principle. Under this equivalence, the standard neural network models can be viewed as scalar (single sample) IB quantizers. It is known, from conventional rate-distortion theory, that scalar quantizers are inferior to vector (multi-sample) quantizers. Such a deficiency then inspires us to develop a novel learning framework, AgrLearn, that corresponds to vector IB quantizers for learning with neural networks. Unlike standard networks, AgrLearn simultaneously optimizes against multiple data samples. We experimentally verify that AgrLearn can result in significant improvements when applied to several current deep learning architectures for image recognition and text classification. We also empirically show that AgrLearn can reduce up to 80% of the training samples needed for ResNet training.
Code (0)
등록된 구현이 없습니다.
Tasks
Image ClassificationQuantizationtext-classificationText ClassificationSimilar Papers 제목 키워드 기반
Aggregated Learning: A Vector-Quantization Approach to Learning Neural Network Classifiers
We consider the problem of learning a neural network classifier. Under the information bottleneck (IB) principle, we associate with this classification problem a representation learning problem, which we call "IB learnin…
ClassificationGeneral ClassificationQuantizationRepresentation Learning+2Gaussian Mixture Vector Quantization with Aggregated Categorical Posterior
The vector quantization is a widely used method to map continuous representation to discrete space and has important application in tokenization for generative mode, bottlenecking information and many other tasks in mach…
QuantizationHACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference
Disaggregated Large Language Model (LLM) inference has gained popularity as it separates the computation-intensive prefill stage from the memory-intensive decode stage, avoiding the prefill-decode interference and improv…
Language ModelingLanguage ModellingLarge Language ModelQuantizationBandwidth-Efficient Multi-Agent Communication through Information Bottleneck and Vector Quantization
Multi-agent reinforcement learning systems deployed in real-world robotics applications face severe communication constraints that significantly impact coordination effectiveness. We present a framework that combines inf…
Multi-agent Reinforcement LearningRQ-MoE: Residual Quantization via Mixture of Experts for Efficient Input-Dependent Vector Compression
Vector quantization is a fundamental tool for compressing high-dimensional embeddings, yet existing multi-codebook methods rely on static codebooks that limit expressiveness under heterogeneous data geometry. While recen…