Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
Gradient-Free Structured Pruning with Unlabeled Data
Large Language Models (LLMs) have achieved great success in solving difficult tasks across many domains, but such success comes with a high computation cost, and inference latency. As developers and third parties customi…
GPUModel CompressionAdversarial Attacks on Machine Learning in Embedded and IoT Platforms
Machine learning (ML) algorithms are increasingly being integrated into embedded and IoT systems that surround us, and they are vulnerable to adversarial attacks. The deployment of these ML algorithms on resource-limited…
Adversarial RobustnessModel CompressionRotation Invariant Quantization for Model Compression
Post-training Neural Network (NN) model compression is an attractive approach for deploying large, memory-consuming models on devices with limited memory resources. In this study, we investigate the rate-distortion trade…
modelModel CompressionQuantizationTowards domain generalisation in ASR with elitist sampling and ensemble knowledge distillation
Knowledge distillation has widely been used for model compression and domain adaptation for speech applications. In the presence of multiple teachers, knowledge can easily be transferred to the student by averaging the m…
Domain AdaptationKnowledge DistillationModel CompressionStructured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding
Self-supervised speech representation learning (SSL) has shown to be effective in various downstream tasks, but SSL models are usually large and slow. Model compression techniques such as pruning aim to reduce the model …
Model CompressionRepresentation Learningspeech-recognitionSpeech Recognition+1Debiased Distillation by Transplanting the Last Layer
Deep models are susceptible to learning spurious correlations, even during the post-processing. We take a closer look at the knowledge distillation -- a popular post-processing technique for model compression -- and find…
AttributeKnowledge DistillationModel CompressionStructured Bayesian Compression for Deep Neural Networks Based on The Turbo-VBI Approach
With the growth of neural network size, model compression has attracted increasing interest in recent research. As one of the most common techniques, pruning has been studied for a long time. By exploiting the structured…
Model CompressionHomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers
Knowledge distillation has been shown to be a powerful model compression approach to facilitate the deployment of pre-trained language models in practice. This paper focuses on task-agnostic distillation. It produces a c…
Knowledge DistillationModel CompressionTransfer LearningA Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
The union of Edge Computing (EC) and Artificial Intelligence (AI) has brought forward the Edge AI concept to provide intelligent solutions close to the end-user environment, for privacy preservation, low latency to real-…
Edge-computingModel CompressionTowards Optimal Compression: Joint Pruning and Quantization
Model compression is instrumental in optimizing deep neural network inference on resource-constrained hardware. The prevailing methods for network compression, namely quantization and pruning, have been shown to enhance …
Model CompressionNeural Architecture SearchQuantizationDual Relation Knowledge Distillation for Object Detection
Knowledge distillation is an effective method for model compression. However, it is still a challenging topic to apply knowledge distillation to detection tasks. There are two key points resulting in poor distillation pe…
Knowledge DistillationModel CompressionObjectobject-detection+2On Achieving Privacy-Preserving State-of-the-Art Edge Intelligence
Deep Neural Network (DNN) Inference in Edge Computing, often called Edge Intelligence, requires solutions to insure that sensitive data confidentiality and intellectual property are not revealed in the process. Privacy-p…
Edge-computingModel CompressionPositionPrivacy PreservingKnowledge Distillation in Vision Transformers: A Critical Review
In Natural Language Processing (NLP), Transformers have already revolutionized the field by utilizing an attention-based encoder-decoder model. Recently, some pioneering works have employed Transformer-like architectures…
Decoderimage-classificationImage ClassificationKnowledge Distillation+5Generalized Uncertainty of Deep Neural Networks: Taxonomy and Applications
Deep neural networks have seen enormous success in various real-world applications. Beyond their predictions as point estimates, increasing attention has been focused on quantifying the uncertainty of their predictions. …
Knowledge DistillationModel CompressionUncertainty QuantificationWeakly-supervised LearningKnowledge Distillation on Graphs: A Survey
Graph Neural Networks (GNNs) have attracted tremendous attention by demonstrating their capability to handle graph data. However, they are difficult to be deployed in resource-limited devices due to model sizes and scala…
Knowledge DistillationModel CompressionSurveyUPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers
Real-world data contains a vast amount of multimodal information, among which vision and language are the two most representative modalities. Moreover, increasingly heavier models, \textit{e}.\textit{g}., Transformers, h…
Image CaptioningImage ClassificationImage RetrievalImage-text Retrieval+7AMD: Adaptive Masked Distillation for Object Detection
As a general model compression paradigm, feature-based knowledge distillation allows the student model to learn expressive features from the teacher counterpart. In this paper, we mainly focus on designing an effective f…
Knowledge DistillationModel CompressionObjectobject-detection+1Improved knowledge distillation by utilizing backward pass knowledge in neural networks
Knowledge distillation (KD) is one of the prominent techniques for model compression. In this method, the knowledge of a large network (teacher) is distilled into a model (student) with usually significantly fewer parame…
Knowledge DistillationModel CompressionHALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks
Low-rank compression is an important model compression strategy for obtaining compact neural network models. In general, because the rank values directly determine the model complexity and model accuracy, proper selectio…
GPULow-rank compressionModel CompressionAccelerating and Compressing Deep Neural Networks for Massive MIMO CSI Feedback
The recent advances in machine learning and deep neural networks have made them attractive candidates for wireless communications functions such as channel estimation, decoding, and downlink channel state information (CS…
Model CompressionNetwork PruningQuantization