Neural Weight Compression for Language Models
Efficient compression of language model weights is increasingly critical as model scale and deployment grow. Yet, most existing methods rely on handcrafted transforms and heuristics, reflecting the limited understanding of weights as a data modality. To move beyond this paradigm, we formulate weight compression as neural codec learning and propose Neural Weight Compression (NWC), a framework for training neural codecs on pretrained weight datasets. NWC addresses challenges intrinsic to weight compression, including tensor heterogeneity and the mismatch between reconstruction losses and downstream performance. Experiments show that NWC achieves highly competitive accuracy-compression tradeoffs, with particularly strong results in the 4-6 bit regime, without relying on rigid handcrafted components such as the Hadamard transform. These gains extend to across diverse architectures, e.g., vision encoders. Our analysis highlights the roles of entropy-constrained quantization and learned transforms in adapting compression to weight data and downstream tasks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression
Despite significant advancements, the practical deployment of Large Language Models (LLMs) is often hampered by their immense sizes, highlighting the need for effective compression techniques. Singular Value Decompositio…
Language ModelingLanguage ModellingLarge Language ModelModel CompressionPC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillation
Low-rank adaption (LoRA) is a prominent method that adds a small number of learnable parameters to the frozen pre-trained weights for parameter-efficient fine-tuning. Prompted by the question, ``Can we make its represent…
Knowledge DistillationModel Compressionparameter-efficient fine-tuningLVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compression
Despite recent efforts in understanding the compression impact on large language models (LLMs) in terms of their downstream task performance and trustworthiness on relatively simpler uni-modal benchmarks (for example, qu…
BenchmarkingCommon Sense ReasoningHallucinationLanguage Modeling+6PocketLLM: Ultimate Compression of Large Language Models via Meta Networks
As Large Language Models (LLMs) continue to grow in size, storing and transmitting them on edge devices becomes increasingly challenging. Traditional methods like quantization and pruning struggle to achieve extreme comp…
CASP: Compression of Large Multimodal Models Based on Attention Sparsity
In this work, we propose an extreme compression technique for Large Multimodal Models (LMMs). While previous studies have explored quantization as an efficient post-training compression method for Large Language Models (…
Model CompressionQuantization