Low-Rank Matrix Approximation for Neural Network Compression
Deep Neural Networks (DNNs) have encountered an emerging deployment challenge due to large and expensive memory and computation requirements. In this paper, we present a new Adaptive-Rank Singular Value Decomposition (ARSVD) method that approximates the optimal rank for compressing weight matrices in neural networks using spectral entropy. Unlike conventional SVD-based methods that apply a fixed-rank truncation across all layers, ARSVD uses an adaptive selection of the rank per layer through the entropy distribution of its singular values. This approach ensures that each layer will retain a certain amount of its informational content, thereby reducing redundancy. Our method enables efficient, layer-wise compression, yielding improved performance with reduced space and time complexity compared to static-rank reduction techniques.
Code (0)
등록된 구현이 없습니다.
Tasks
Model CompressionNeural Network CompressionSimilar Papers 제목 키워드 기반
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
Transformer models have achieved remarkable results in various natural language tasks, but they are often prohibitively large, requiring massive memories and computational resources. To reduce the size and complexity of …
DiversityModel CompressionNatural Language UnderstandingQuestion Answering+1Reweighted Solutions for Weighted Low Rank Approximation
Weighted low rank approximation (WLRA) is an important yet computationally challenging primitive with applications ranging from statistical analysis, model compression, and signal processing. To cope with the NP-hardness…
feature selectionModel CompressionCompression of Recurrent Neural Networks using Matrix Factorization
Compressing neural networks is a key step when deploying models for real-time or embedded applications. Factorizing the model's matrices using low-rank approximations is a promising method for achieving compression. Whil…
Singular Value Decomposition of Images from Scanned Photographic Plates
We want to approximate the mxn image A from scanned astronomical photographic plates (from the Sofia Sky Archive Data Center) by using far fewer entries than in the original matrix. By using rank of a matrix, k we remove…
Image CompressionEfficient Wireless Federated Learning via Low-Rank Gradient Factorization
This paper presents a novel gradient compression method for federated learning (FL) in wireless systems. The proposed method centers on a low-rank matrix factorization strategy for local gradient compression that is base…
Federated LearningLow-rank compression