Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases
The deployment of Large Language Models (LLMs) and Large Multimodal Models (LMMs) on mobile devices has gained significant attention due to the benefits of enhanced privacy, stability, and personalization. However, the h…
BenchmarkingModel CompressionQuantizationOn the social bias of speech self-supervised models
Self-supervised learning (SSL) speech models have achieved remarkable performance in various tasks, yet the biased outcomes, especially affecting marginalized groups, raise significant concerns. Social bias refers to the…
Model CompressionSelf-Supervised LearningEnhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspective
Pre-trained large language models (LLMs) based on Transformer have demonstrated striking in-context learning (ICL) abilities. With a few demonstration input-label pairs, they can predict the label for an unseen input wit…
Generalization BoundsIn-Context LearningModel CompressionSlicing Mutual Information Generalization Bounds for Neural Networks
The ability of machine learning (ML) algorithms to generalize well to unseen data has been studied through the lens of information theory, by bounding the generalization error with the input-output mutual information (MI…
Generalization BoundsModel CompressionReweighted Solutions for Weighted Low Rank Approximation
Weighted low rank approximation (WLRA) is an important yet computationally challenging primitive with applications ranging from statistical analysis, model compression, and signal processing. To cope with the NP-hardness…
feature selectionModel CompressionTowards Efficient Deep Spiking Neural Networks Construction with Spiking Activity based Pruning
The emergence of deep and large-scale spiking neural networks (SNNs) exhibiting high performance across diverse complex datasets has led to a need for compressing network models due to the presence of a significant numbe…
Model CompressionNetwork PruningSparse LearningRobust Knowledge Distillation Based on Feature Variance Against Backdoored Teacher Model
Benefiting from well-trained deep neural networks (DNNs), model compression have captured special attention for computing resource limited equipment, especially edge devices. Knowledge distillation (KD) is one of the wid…
Knowledge DistillationModel CompressionEffective Interplay between Sparsity and Quantization: From Theory to Practice
The increasing size of deep neural networks (DNNs) necessitates effective model compression to reduce their computational and memory footprints. Sparsity and quantization are two prominent compression methods that have b…
Computational EfficiencyModel CompressionQuantizationLCQ: Low-Rank Codebook based Quantization for Large Language Models
Large language models~(LLMs) have recently demonstrated promising performance in many tasks. However, the high storage and computational cost of LLMs has become a challenge for deploying LLMs. Weight quantization has bee…
Model CompressionQuantizationDual sparse training framework: inducing activation map sparsity via Transformed $\ell1$ regularization
Although deep convolutional neural networks have achieved rapid development, it is challenging to widely promote and apply these models on low-power devices, due to computational and storage limitations. To address this …
Model CompressionOccam Gradient Descent
Deep learning neural network models must be large enough to adapt to their problem domain, while small enough to avoid overfitting training data during gradient descent. To balance these competing demands, overprovisione…
image-classificationImage ClassificationLearning TheoryModel CompressionsubMFL: Compatiple subModel Generation for Federated Learning in Device Heterogenous Environment
Federated Learning (FL) is commonly used in systems with distributed and heterogeneous devices with access to varying amounts of data and diverse computing and storage capacities. FL training process enables such devices…
Federated LearningModel CompressionNV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
Decoder-only LLM-based embedding models are beginning to outperform BERT or T5-based embedding models in general-purpose text embedding tasks, including dense vector-based retrieval. In this work, we introduce NV-Embed, …
Information RetrievalLanguage ModellingLarge Language ModelModel Compression+4ExtremeMETA: High-speed Lightweight Image Segmentation Model by Remodeling Multi-channel Metamaterial Imagers
Deep neural networks (DNNs) have heavily relied on traditional computational units like CPUs and GPUs. However, this conventional approach brings significant computational burdens, latency issues, and high power consumpt…
Image SegmentationModel CompressionSegmentationSemantic SegmentationEfficient Model Compression for Hierarchical Federated Learning
Federated learning (FL), as an emerging collaborative learning paradigm, has garnered significant attention due to its capacity to preserve privacy within distributed learning systems. In these systems, clients collabora…
Edge-computingFederated LearningmodelModel CompressionEfficiency optimization of large-scale language models based on deep learning in natural language processing tasks
The internal structure and operation mechanism of large-scale language models are analyzed theoretically, especially how Transformer and its derivative architectures can restrict computing efficiency while capturing long…
Inference OptimizationKnowledge DistillationModel CompressionTinyM$^2$Net-V3: Memory-Aware Compressed Multimodal Deep Neural Networks for Sustainable Edge Deployment
The advancement of sophisticated artificial intelligence (AI) algorithms has led to a notable increase in energy usage and carbon dioxide emissions, intensifying concerns about climate change. This growing problem has br…
Knowledge DistillationModel CompressionQuantizationDensely Distilling Cumulative Knowledge for Continual Learning
Continual learning, involving sequential training on diverse tasks, often faces catastrophic forgetting. While knowledge distillation-based approaches exhibit notable success in preventing forgetting, we pinpoint a limit…
AllContinual LearningKnowledge DistillationModel CompressionAdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weighting
Knowledge distillation, a widely used model compression technique, works on the basis of transferring knowledge from a cumbersome teacher model to a lightweight student model. The technique involves jointly optimizing th…
Knowledge DistillationModel CompressionCharacterizing the Accuracy -- Efficiency Trade-off of Low-rank Decomposition in Language Models
Recent large language models (LLMs) employ billions of parameters to enable broad problem-solving capabilities. Such language models also tend to be memory-bound because of the dominance of matrix-vector and matrix-matri…
AI AgentModel CompressionNavigateQuantization