Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation
Data-free knowledge distillation (DFKD) has emerged as a pivotal technique in the domain of model compression, substantially reducing the dependency on the original training data. Nonetheless, conventional DFKD methods t…
Data-free Knowledge DistillationDiversityKnowledge DistillationModel CompressionSelf-calibration for Language Model Quantization and Pruning
Quantization and pruning are fundamental approaches for model compression, enabling efficient inference for language models. In a post-training setting, state-of-the-art quantization and pruning methods require calibrati…
Language ModelingLanguage ModellingmodelModel Compression+1Identifying Sub-networks in Neural Networks via Functionally Similar Representations
Providing human-understandable insights into the inner workings of neural networks is an important step toward achieving more explainable and trustworthy AI. Existing approaches to such mechanistic interpretability typic…
Model CompressionPreview-based Category Contrastive Learning for Knowledge Distillation
Knowledge distillation is a mainstream algorithm in model compression by transferring knowledge from the larger model (teacher) to the smaller model (student) to improve the performance of student. Despite many efforts, …
Contrastive LearningKnowledge DistillationModel CompressionEvoPress: Towards Optimal Dynamic Model Compression via Evolutionary Search
The high computational costs of large language models (LLMs) have led to a flurry of research on LLM compression, via methods such as quantization, sparsification, or structured pruning. A new frontier in this area is gi…
Model CompressionQuantizationQIANets: Quantum-Integrated Adaptive Networks for Reduced Latency and Improved Inference Times in CNN Models
Convolutional neural networks (CNNs) have made significant advances in computer vision tasks, yet their high inference times and latency often limit real-world applicability. While model compression techniques have gaine…
Model CompressionTensor DecompositionSLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
Conventional model compression techniques for LLMs address high memory consumption and slow inference challenges but typically require computationally expensive retraining to preserve accuracy. In contrast, one-shot comp…
Model CompressionNatural Language Understandingparameter-efficient fine-tuningQuantizationCrossQuant: A Post-Training Quantization Method with Smaller Quantization Kernel for Precise Large Language Model Compression
Post-Training Quantization (PTQ) is an effective technique for compressing Large Language Models (LLMs). While many studies focus on quantizing both weights and activations, it is still a challenge to maintain the accura…
Language ModelingLanguage ModellingLarge Language ModelModel Compression+1What is Left After Distillation? How Knowledge Transfer Impacts Fairness and Bias
Knowledge Distillation is a commonly used Deep Neural Network (DNN) compression method, which often maintains overall generalization performance. However, we show that even for balanced image classification datasets, suc…
Age/UnbiasedFairnessGender Bias Detectionimage-classification+6Large Language Model Compression with Neural Architecture Search
Large language models (LLMs) exhibit remarkable reasoning abilities, allowing them to generalize across a wide range of downstream tasks, such as commonsense reasoning or instruction following. However, as LLMs scale, in…
Instruction FollowingLanguage ModelingLanguage ModellingLarge Language Model+4QT-DoG: Quantization-aware Training for Domain Generalization
Domain Generalization (DG) aims to train models that perform well not only on the training (source) domains but also on novel, unseen target data distributions. A key challenge in DG is preventing overfitting to source d…
Domain GeneralizationModel CompressionQuantizationSpaLLM: Unified Compressive Adaptation of Large Language Models with Sketching
Compressive adaptation approaches, such as QLoRA, are widely popular alternatives for reducing memory requirements during fine-tuning of large language models (LLMs) while producing models capable of handling various dow…
Model CompressionNatural Language UnderstandingESPACE: Dimensionality Reduction of Activations for Model Compression
We propose ESPACE, an LLM compression technique based on dimensionality reduction of activations. Unlike prior works on weight-centric tensor decomposition, ESPACE projects activations onto a pre-calibrated set of princi…
Dimensionality ReductionmodelModel CompressionTensor DecompositionContinuous Approximations for Improving Quantization Aware Training of LLMs
Model compression methods are used to reduce the computation and energy requirements for Large Language Models (LLMs). Quantization Aware Training (QAT), an effective model compression method, is proposed to reduce perfo…
MMLUModel CompressionQuantizationGeometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models
Matrix and tensor-guided parametrization for Natural Language Processing (NLP) models is fundamentally useful for the improvement of the model's systematic efficiency. However, the internal links between these two algebr…
AllLanguage ModelingLanguage ModellingMath+2Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression
Large Language Models (LLMs) have achieved remarkable breakthroughs. However, the huge number of parameters in LLMs require significant amount of memory storage in inference, which prevents their practical deployment in …
Language ModelingLanguage ModellingLarge Language ModelModel CompressionTrainable pruned ternary quantization for medical signal classification models
The field of deep learning is renowned for its resource-intensive nature, hence improving its environmental impact is crucial. In this paper, we propose a novel model compression method to mitigate the energy demands of …
Model CompressionQuantizationCompressing Recurrent Neural Networks for FPGA-accelerated Implementation in Fluorescence Lifetime Imaging
Fluorescence lifetime imaging (FLI) is an important technique for studying cellular environments and molecular interactions, but its real-time application is limited by slow data acquisition, which requires capturing lar…
Computational EfficiencyKnowledge DistillationModel CompressionQuantizationAggressive Post-Training Compression on Extremely Large Language Models
The increasing size and complexity of Large Language Models (LLMs) pose challenges for their deployment on personal computers and mobile devices. Aggressive post-training model compression is necessary to reduce the mode…
Model CompressionNetwork PruningQuantizationInfantCryNet: A Data-driven Framework for Intelligent Analysis of Infant Cries
Understanding the meaning of infant cries is a significant challenge for young parents in caring for their newborns. The presence of background noise and the lack of labeled data present practical challenges in developin…
Knowledge DistillationModel CompressionModel SelectionQuantization