Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition
High-dimensional token embeddings underpin Large Language Models (LLMs), as they can capture subtle semantic information and significantly enhance the modelling of complex language patterns. However, this high dimensiona…
Model CompressionQuantization Variation: A New Perspective on Training Transformers with Low-Bit Precision
Despite the outstanding performance of transformers in both language and vision tasks, the expanding computation and model size have increased the demand for efficient deployment. To address the heavy computation and par…
Knowledge DistillationModel CompressionQuantizationAn Efficient Sparse Inference Software Accelerator for Transformer-based Language Models on CPUs
In recent years, Transformer-based language models have become the standard approach for natural language processing tasks. However, stringent throughput and latency requirements in industrial applications are limiting t…
Model CompressionConstraint-aware and Ranking-distilled Token Pruning for Efficient Transformer Inference
Deploying pre-trained transformer models like BERT on downstream tasks in resource-constrained scenarios is challenging due to their high inference cost, which grows rapidly with input sequence length. In this work, we p…
CPUModel CompressionFeature Adversarial Distillation for Point Cloud Classification
Due to the point cloud's irregular and unordered geometry structure, conventional knowledge distillation technology lost a lot of information when directly used on point cloud tasks. In this paper, we propose Feature Adv…
ClassificationFADKnowledge DistillationModel Compression+2Low-Rank Prune-And-Factorize for Language Model Compression
The components underpinning PLMs -- large weight matrices -- were shown to bear considerable redundancy. Matrix factorization, a well-established technique from matrix theory, has been utilized to reduce the number of pa…
Language ModelingLanguage ModellingModel CompressionNetwork Pruning+1Partitioning-Guided K-Means: Extreme Empty Cluster Resolution for Extreme Model Compression
Compactness in deep learning can be critical to a model's viability in low-resource applications, and a common approach to extreme model compression is quantization. We consider Iterative Product Quantization (iPQ) with …
Model CompressionQuantizationData-Free Backbone Fine-Tuning for Pruned Neural Networks
Model compression techniques reduce the computational load and memory consumption of deep neural networks. After the compression operation, e.g. parameter pruning, the model is normally fine-tuned on the original trainin…
2D Human Pose Estimationimage-classificationImage ClassificationModel Compression+3CrossKD: Cross-Head Knowledge Distillation for Object Detection
Knowledge Distillation (KD) has been validated as an effective model compression technique for learning compact object detectors. Existing state-of-the-art KD methods for object detection are mostly based on feature imit…
Dense Object DetectionKnowledge DistillationModel CompressionObject+2LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation
Transformer models have achieved remarkable results in various natural language tasks, but they are often prohibitively large, requiring massive memories and computational resources. To reduce the size and complexity of …
DiversityModel CompressionNatural Language UnderstandingQuestion Answering+1DynaQuant: Compressing Deep Learning Training Checkpoints via Dynamic Quantization
With the increase in the scale of Deep Learning (DL) training workloads in terms of compute resources and time consumption, the likelihood of encountering in-training failures rises substantially, leading to lost work an…
Deep LearningModel CompressionQuantizationTransfer LearningHiNeRV: Video Compression with Hierarchical Encoding-based Neural Representation
Learning-based video compression is currently a popular research topic, offering the potential to compete with conventional standard video codecs. In this context, Implicit Neural Representations (INRs) have previously b…
Model CompressionQuantizationVideo CompressionNeural Network Compression using Binarization and Few Full-Precision Weights
Quantization and pruning are two effective Deep Neural Networks model compression methods. In this paper, we propose Automatic Prune Binarization (APB), a novel compression technique combining quantization with pruning. …
BinarizationCPUModel CompressionNeural Network Compression+1Efficient and Robust Quantization-aware Training via Adaptive Coreset Selection
Quantization-aware training (QAT) is a representative model compression method to reduce redundancy in weights and activations. However, most existing QAT methods require end-to-end training on the entire dataset, which …
Model CompressionQuantizationDeep Model Compression Also Helps Models Capture Ambiguity
Natural language understanding (NLU) tasks face a non-trivial amount of ambiguous samples where veracity of their labels is debatable among annotators. NLU models should thus account for such ambiguity, but they approxim…
modelModel CompressionNatural Language UnderstandingA Brief Review of Hypernetworks in Deep Learning
Hypernetworks, or hypernets for short, are neural networks that generate weights for another neural network, known as the target network. They have emerged as a powerful deep learning technique that allows for greater fl…
Causal InferenceContinual LearningDeep LearningModel Compression+3MobileNMT: Enabling Translation in 15MB and 30ms
Deploying NMT models on mobile devices is essential for privacy, low latency, and offline scenarios. For high model capacity, NMT models are rather large. Running these models on devices is challenging with limited stora…
Model CompressionNMTQuantizationTranslationRiemannian Low-Rank Model Compression for Federated Learning with Over-the-Air Aggregation
Low-rank model compression is a widely used technique for reducing the computational load when training machine learning models. However, existing methods often rely on relaxing the low-rank constraint of the model weigh…
Federated LearningModel CompressionRiemannian optimizationModular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference
Pre-trained Transformer models like T5 and BART have advanced the state of the art on a wide range of text generation tasks. Compressing these models into smaller ones has become critically important for practical use. C…
DecoderKnowledge DistillationModel CompressionNeural Network Compression+2Low-Complexity Acoustic Scene Classification Using Data Augmentation and Lightweight ResNet
We present a work on low-complexity acoustic scene classification (ASC) with multiple devices, namely the subtask A of Task 1 of the DCASE2021 challenge. This subtask focuses on classifying audio samples of multiple devi…
Acoustic Scene ClassificationData AugmentationModel CompressionScene Classification