Papers Neural Network Compression
“Neural Network Compression” 태그가 달린 논문 216편 · 필터 해제
Multi-head Knowledge Distillation for Model Compression
Several methods of knowledge distillation have been developed for neural network compression. While they all use the KL divergence loss to align the soft outputs of the student model more closely with that of the teacher…
image-classificationImage ClassificationKnowledge Distillationmodel+2Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression
Deep neural networks (DNNs) have been extremely successful in solving many challenging AI tasks in natural language processing, speech recognition, and computer vision nowadays. However, DNNs are typically computation in…
Knowledge DistillationNeural Network CompressionQuantizationspeech-recognition+1torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation
While knowledge distillation (transfer) has been attracting attentions from the research community, the recent development in the fields has heightened the need for reproducible studies and highly generalized frameworks …
Image ClassificationInstance SegmentationKnowledge DistillationModel Compression+2Head Network Distillation: Splitting Distilled Deep Neural Networks for Resource-Constrained Edge Computing Systems
As the complexity of Deep Neural Network (DNN) models increases, their deployment on mobile devices becomes increasingly challenging, especially in complex vision tasks such as image classification. Many of recent contri…
Edge-computingimage-classificationImage ClassificationKnowledge Distillation+2Deep convolutional neural network compression via coupled tensor decomposition
Large neural networks have aroused impressive progress in various real world applications. However, the expensive storage and computational resources requirement for running deep networks make them problematic to be depl…
Image ReconstructionNeural Network CompressionTensor DecompositionNeural Network Compression Via Sparse Optimization
The compression of deep neural networks (DNNs) to reduce inference cost becomes increasingly important to meet realistic deployment requirements of various applications. There have been a significant amount of work regar…
Model CompressionNeural Network CompressionStochastic OptimizationDirichlet Pruning for Neural Network Compression
We introduce Dirichlet pruning, a novel post-processing technique to transform a large neural network model into a compressed one. Dirichlet pruning is a form of structured pruning that assigns the Dirichlet distribution…
Neural Network CompressionVariational InferenceNoisy Neural Network Compression for Analog Storage Devices
Efficient compression and storage of neural network (NN) parameters is critical for resource-constrained, downstream machine learning applications. Although several methods for NN compression have been developed, there h…
Knowledge DistillationModel CompressionNeural Network CompressionAdditive Tree-Structured Conditional Parameter Spaces in Bayesian Optimization: A Novel Covariance Function and a Fast Implementation
Bayesian optimization (BO) is a sample-efficient global optimization algorithm for black-box functions which are expensive to evaluate. Existing literature on model based optimization in conditional parameter spaces are …
Bayesian Optimizationglobal-optimizationNeural Network CompressionSMAC+1A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions
Deep Neural Network (DNN) has gained unprecedented performance due to its automated feature extraction capability. This high order performance leads to significant incorporation of DNN models in different Internet of Thi…
Knowledge DistillationMiscellaneousNetwork PruningNeural Network CompressionTransform Quantization for CNN (Convolutional Neural Network) Compression
In this paper, we compress convolutional neural network (CNN) weights post-training via transform quantization. Previous CNN quantization techniques tend to ignore the joint statistics of weights and activations, produci…
Dimensionality ReductionNeural Network CompressionQuantizationDifferentiable Joint Pruning and Quantization for Hardware Efficiency
We present a differentiable joint pruning and quantization (DJPQ) scheme. We frame neural network compression as a joint gradient-based optimization problem, trading off between model pruning and quantization automatical…
Neural Network CompressionQuantizationT-Basis: a Compact Representation for Neural Networks
We introduce T-Basis, a novel concept for a compact representation of a set of tensors, each of an arbitrary shape, which is often seen in Neural Networks. Each of the tensors in the set is modeled using Tensor Rings, th…
Neural Network CompressionTensor NetworksHybrid Tensor Decomposition in Neural Network Compression
Deep neural networks (DNNs) have enabled impressive breakthroughs in various artificial intelligence (AI) applications recently due to its capability of learning high-level features from big data. However, the current de…
Neural Network CompressionTensor DecompositionNeural gradients are near-lognormal: improved quantized and sparse training
While training can mostly be accelerated by reducing the time needed to propagate neural gradients back throughout the model, most previous works focus on the quantization/pruning of weights and activations. These method…
Neural Network CompressionQuantizationNeural Network Compression Using Higher-Order Statistics and AuxiliaryReconstruction Losses
In this paper, the problem of pruning and compressingthe weights of various layers of deep neural networks is in-vestigated. The proposed method aims to remove redundantfilters from the network to reduce computational co…
Neural Network CompressionConvolutional neural networks compression with low rank and sparse tensor decompositions
Convolutional neural networks show outstanding results in a variety of computer vision tasks. However, a neural network architecture design usually faces a trade-off between model performance and computational/memory com…
CPUimage-classificationImage ClassificationNeural Network CompressionAn Overview of Neural Network Compression
Overparameterized networks trained to convergence have shown impressive performance in domains such as computer vision and natural language processing. Pushing state of the art on salient tasks within these domains corre…
Knowledge DistillationModel CompressionNeural Network CompressionQuantization+1Learning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration
Filter pruning has been widely applied to neural network compression and acceleration. Existing methods usually utilize pre-defined pruning criteria, such as Lp-norm, to prune unimportant filters. There are two major lim…
image-classificationImage ClassificationNeural Network CompressionWoodFisher: Efficient Second-Order Approximation for Neural Network Compression
Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been significant interest in utilizing this information i…
image-classificationImage ClassificationModel CompressionNeural Network Compression