NETWORK COMPRESSION USING CORRELATION ANALYSIS OF LAYER RESPONSES
Principal Filter Analysis (PFA) is an easy to implement, yet effective method for neural network compression. PFA exploits the intrinsic correlation between filter responses within network layers to recommend a smaller network footprint. We propose two compression algorithms: the first allows a user to specify the proportion of the original spectral energy that should be preserved in each layer after compression, while the second is a heuristic that leads to a parameter-free approach that automatically selects the compression used at each layer. Both algorithms are evaluated against several architectures and datasets, and we show considerable compression rates without compromising accuracy, e.g., for VGG-16 on CIFAR-10, CIFAR-100 and ImageNet, PFA achieves a compression rate of 8x, 3x, and 1.4x with an accuracy gain of 0.4%, 1.4% points, and 2.4% respectively. In our tests we also demonstrate that networks compressed with PFA achieve an accuracy that is very close to the empirical upper bound for a given compression ratio. Finally, we show how PFA is an effective tool for simultaneous compression and domain adaptation.
Code (0)
등록된 구현이 없습니다.
Tasks
Domain AdaptationNeural Network CompressionSimilar Papers 제목 키워드 기반
Filter Distillation for Network Compression
In this paper we introduce Principal Filter Analysis (PFA), an easy to use and effective method for neural network compression. PFA exploits the correlation between filter responses within network layers to recommend a s…
Domain AdaptationNeural Network CompressionPhilosophyImplicit Compression Regularization: Concise Reasoning via Internal Shorter Distributions in RL Post-Training
Reinforcement learning with verifiable rewards improves LLM reasoning but often induces overthinking, where models generate unnecessarily long reasoning traces. Existing methods mainly rely on length penalties or early-e…
Reinforcement LearningHow Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?
Tensor decompositions have been successfully applied to compress neural networks. The compression algorithms using tensor decompositions commonly minimize the approximation error on the weights. Recent work assumes the a…
Neural Network CompressionTensor DecompositionProcessing Energy Modeling for Neural Network Based Image Compression
Nowadays, the compression performance of neural-networkbased image compression algorithms outperforms state-of-the-art compression approaches such as JPEG or HEIC-based image compression. Unfortunately, most neural-netwo…
GPUImage CompressionCPTQuant -- A Novel Mixed Precision Post-Training Quantization Techniques for Large Language Models
Large language models have transformed the comprehension and generation of natural language tasks, but they come with substantial memory and computational requirements. Quantization techniques have emerged as a promising…
Language ModelingLanguage ModellingModel CompressionQuantization+1