Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
Group channel pruning and spatial attention distilling for object detection
Due to the over-parameterization of neural networks, many model compression methods based on pruning and quantization have emerged. They are remarkable in reducing the size, parameter number, and computational complexity…
Knowledge DistillationModel Compressionobject-detectionObject Detection+1Task-Agnostic Structured Pruning of Speech Representation Models
Self-supervised pre-trained models such as Wav2vec2, Hubert, and WavLM have been shown to significantly improve many speech tasks. However, their large memory and strong computational requirements hinder their industrial…
Model CompressionConaCLIP: Exploring Distillation of Fully-Connected Knowledge Interaction Graph for Lightweight Text-Image Retrieval
Large-scale pre-trained text-image models with dual-encoder architectures (such as CLIP) are typically adopted for various vision-language applications, including text-image retrieval. However,these models are still less…
Image RetrievalKnowledge DistillationModel CompressionRetrievalLoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning
Large Language Models (LLMs), such as LLaMA and T5, have shown exceptional performance across various tasks through fine-tuning. Although low-rank adaption (LoRA) has emerged to cheaply fine-tune these LLMs on downstream…
Model CompressionNetwork Pruningparameter-efficient fine-tuning2-bit Conformer quantization for automatic speech recognition
Large speech models are rapidly gaining traction in research community. As a result, model compression has become an important topic, so that these models can fit in memory and be served with reduced cost. Practical appr…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Model CompressionQuantization+2COMCAT: Towards Efficient Compression and Customization of Attention-Based Vision Models
Attention-based vision models, such as Vision Transformer (ViT) and its variants, have shown promising performance in various computer vision tasks. However, these emerging architectures suffer from large model sizes and…
Model CompressionPruMUX: Augmenting Data Multiplexing with Model Compression
As language models increase in size by the day, methods for efficient inference are critical to leveraging their capabilities for various applications. Prior work has investigated techniques like model pruning, knowledge…
Knowledge DistillationmodelModel CompressionAn Efficient Multilingual Language Model Compression through Vocabulary Trimming
Multilingual language model (LM) have become a powerful tool in NLP especially for non-English languages. Nevertheless, model parameters of multilingual LMs remain large due to the larger embedding matrix of the vocabula…
Language ModelingLanguage ModellingModel CompressionRAND: Robustness Aware Norm Decay For Quantized Seq2seq Models
With the rapid increase in the size of neural networks, model compression has become an important area of research. Quantization is an effective technique at decreasing the model size, memory access, and compute load of …
Machine TranslationModel CompressionQuantizationspeech-recognition+1Selective Pre-training for Private Fine-tuning
Text prediction models, when used in applications like email clients or word processors, must protect user data privacy and adhere to model size constraints. These constraints are crucial to meet memory and inference tim…
Model CompressionTransfer LearningRevisiting Data Augmentation in Model Compression: An Empirical and Comprehensive Study
The excellent performance of deep neural networks is usually accompanied by a large number of parameters and computations, which have limited their usage on the resource-limited edge devices. To address this issue, abund…
Data AugmentationKnowledge DistillationModel CompressionQuantizationAD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression
Knowledge distillation has attracted a great deal of interest recently to compress pre-trained language models. However, existing knowledge distillation methods suffer from two limitations. First, the student model simpl…
Knowledge DistillationLanguage ModelingLanguage ModellingModel Compression+1Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt
While the numerous parameters in Large Language Models (LLMs) contribute to their superior performance, this massive scale makes them inefficient and memory-hungry. Thus, they are hard to deploy on commodity hardware, su…
GPUModel CompressionPrompt LearningQuantizationTowards Understanding and Improving Knowledge Distillation for Neural Machine Translation
Knowledge distillation (KD) is a promising technique for model compression in neural machine translation. However, where the knowledge hides in KD is still not clear, which may hinder the development of KD. In this work,…
Knowledge DistillationMachine TranslationModel CompressionTranslationGSB: Group Superposition Binarization for Vision Transformer with Limited Training Samples
Vision Transformer (ViT) has performed remarkably in various computer vision tasks. Nonetheless, affected by the massive amount of parameters, ViT usually suffers from serious overfitting problems with a relatively limit…
BinarizationKnowledge DistillationModel CompressionQuantizationCrAFT: Compression-Aware Fine-Tuning for Efficient Visual Task Adaptation
Transfer learning has become a popular task adaptation method in the era of foundation models. However, many foundation models require large storage and computing resources, which makes off-the-shelf deployment impractic…
GPUModel CompressionQuantizationTransfer LearningRedundancy and Concept Analysis for Code-trained Language Models
Code-trained language models have proven to be highly effective for various code intelligence tasks. However, they can be challenging to train and deploy for many software engineering applications due to computational bo…
MemorizationModel CompressionModel OptimizationSentence+2CORSD: Class-Oriented Relational Self Distillation
Knowledge distillation conducts an effective model compression method while holding some limitations:(1) the feature based distillation methods only focus on distilling the feature map but are lack of transferring the re…
Knowledge DistillationModel CompressionRelationRelation ExtractionGuaranteed Quantization Error Computation for Neural Network Model Compression
Neural network model compression techniques can address the computation issue of deep neural networks on embedded devices in industrial systems. The guaranteed output error computation problem for neural network compress…
Model CompressionNeural Network CompressionQuantizationClass Attention Transfer Based Knowledge Distillation
Previous knowledge distillation methods have shown their impressive performance on model compression tasks, however, it is hard to explain how the knowledge they transferred helps to improve the performance of the studen…
Knowledge DistillationModel Compression