Constrained Optimization Based Low-Rank Approximation of Deep Neural Networks
We present COBLA---Constrained Optimization Based Low-rank Approximation---a systematic method of finding an optimal low-rank approximation of a trained convolutional neural network, subject to constraints in the number of multiply-accumulate (MAC) operations and the memory footprint. COBLA optimally allocates the constrained computation resource into each layer of the approximated network. The singular value decomposition of the network weight is computed, then a binary masking variable is introduced to denote whether a particular singular value and the corresponding singular vectors are used in low-rank approximation. With this formulation, the number of the MAC operations and the memory footprint are represented as linear constraints in terms of the binary masking variables. The resulted 0-1 integer programming problem is approximately solved by sequential quadratic programming. COBLA does not introduce any hyperparameter. We empirically demonstrate that COBLA outperforms prior art using the SqueezeNet and VGG-16 architecture on the ImageNet dataset.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Dictionary-based Low-Rank Approximations and the Mixed Sparse Coding problem
Constrained tensor and matrix factorization models allow to extract interpretable patterns from multiway data. Therefore identifiability properties and efficient algorithms for constrained low-rank approximations are now…
Data-driven satisficing measure and ranking
We propose an computational framework for real-time risk assessment and prioritizing for random outcomes without prior information on probability distributions. The basic model is built based on satisficing measure (SM) …
Stochastic Optimization1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models
Large Language Models (LLMs) have demonstrated remarkable proficiency in language comprehension and generation; however, their widespread adoption is constrained by substantial bandwidth and computational demands. While …
MLoRQ: Bridging Low-Rank and Quantization for Transformer Compression
Deploying transformer-based neural networks on resource-constrained edge devices presents a significant challenge. This challenge is often addressed through various techniques, such as low-rank approximation and mixed-pr…
Instance SegmentationImage ClassificationObject DetectionImproved Approximation Algorithms for Low-Rank Problems Using Semidefinite Optimization
Inspired by the impact of the Goemans-Williamson algorithm on combinatorial optimization, we construct an analogous relax-then-sample strategy for low-rank optimization problems. First, for orthogonally constrained quadr…
Combinatorial OptimizationMatrix Completion