Neural Network Compression Using Higher-Order Statistics and AuxiliaryReconstruction Losses
In this paper, the problem of pruning and compressingthe weights of various layers of deep neural networks is in-vestigated. The proposed method aims to remove redundantfilters from the network to reduce computational complex-ity and storage requirements, while improving the perfor-mance of the original network. More specifically, a novelfilter selection criterion is introduced based on the fact thatfilters whose weights follow a Gaussian distribution corre-spond to hidden units that do not capture important aspectsof data. To this end, Higher Order Statistics (HOS) areused and filters with low cumulant values that do not de-viate significantly from Gaussian distribution are identifiedand removed from the network. In addition, a novel prun-ing strategy is proposed aiming to decide on the pruningratio of each layer using the Shapiro-Wilk normality test.The use of auxiliary MSE losses (intermediate and afterthe softmax layer) during the fine-tuning phase further im-proves the overall performance of the compressed network.Extensive experiments with different network architecturesand comparison with state-of-the-art approaches on well-known public datasets, such as CIFAR-10, CIFAR-100 andILSCVR-12, demonstrate the great potential of the proposedapproach.
Code (1)
Tasks
Neural Network CompressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ELiC: Efficient LiDAR Geometry Compression via Cross-Bit-depth Feature Propagation and Bag-of-Encoders
Hierarchical LiDAR geometry compression encodes voxel occupancies from low to high bit-depths, yet prior methods treat each depth independently and re-estimate local context from coordinates at every level, limiting comp…
A Bayesian Optimization Framework for Neural Network Compression
Neural network compression is an important step for deploying neural networks where speed is of high importance, or on devices with limited memory. It is necessary to tune compression parameters in order to achieve the d…
Bayesian OptimizationKnowledge DistillationNeural Network CompressionOn computing and the complexity of computing higher-order $U$-statistics, exactly
Higher-order $U$-statistics abound in fields such as statistics, machine learning, and computer science, but are known to be highly time-consuming to compute in practice. Despite their widespread appearance, a comprehens…
High Order Structure Descriptors for Scene Images
Structure information is ubiquitous in natural scene images and it plays an important role in scene representation. In this paper, third order structure statistics (TOSS) and fourth order structure statistics (FOSS) are …
Scene ClassificationVocal Bursts Intensity PredictionA theory of learning data statistics in diffusion models, from easy to hard
While diffusion models have emerged as a powerful class of generative models, their learning dynamics remain poorly understood. We address this issue first by empirically showing that standard diffusion models trained on…