Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs
Large Language Models (LLMs) have significantly advanced artificial intelligence by optimizing traditional Natural Language Processing (NLP) workflows, facilitating their integration into various systems. Many such NLP s…
Knowledge DistillationModel CompressionQuantizationTransfer LearningImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs
With the proliferation of task-specific large language models, delta compression has emerged as a method to mitigate the resource challenges of deploying numerous such models by effectively compressing the delta model pa…
Model CompressionQuantizationD$^{2}$MoE: Dual Routing and Dynamic Scheduling for Efficient On-Device MoE-based LLM Serving
The mixture of experts (MoE) model is a sparse variant of large language models (LLMs), designed to hold a better balance between intelligent capability and computational overhead. Despite its benefits, MoE is still too …
Mixture-of-ExpertsModel CompressionModel OptimizationQuantization+1Efficient Reasoning Models: A Survey
Reasoning models have demonstrated remarkable progress in solving complex and logic-intensive tasks by generating extended Chain-of-Thoughts (CoTs) prior to arriving at a final answer. Yet, the emergence of this "slow-th…
Knowledge DistillationModel CompressionSurveyEfficient Hybrid Language Model Compression through Group-Aware SSM Pruning
Hybrid LLM architectures that combine Attention and State Space Models (SSMs) achieve state-of-the-art accuracy and runtime performance. Recent work has demonstrated that applying compression and distillation to Attentio…
Knowledge DistillationLanguage ModelingLanguage ModellingModel Compression+1APSQ: Additive Partial Sum Quantization with Algorithm-Hardware Co-Design
DNN accelerators, significantly advanced by model compression and specialized dataflow techniques, have marked considerable progress. However, the frequent access of high-precision partial sums (PSUMs) leads to excessive…
Model CompressionQuantizationTwo is Better than One: Efficient Ensemble Defense for Robust and Compact Models
Deep learning-based computer vision systems adopt complex and large architectures to improve performance, yet they face challenges in deployment on resource-constrained mobile and edge devices. To address this issue, mod…
Adversarial RobustnessDiversityModel CompressionQuantizationThanos: A Block-wise Pruning Algorithm for Efficient Large Language Model Compression
This paper presents Thanos, a novel weight-pruning algorithm designed to reduce the memory footprint and enhance the computational efficiency of large language models (LLMs) by removing redundant weights while maintainin…
Computational EfficiencyLanguage ModelingLanguage ModellingLarge Language Model+1Compression Laws for Large Language Models
We introduce compression laws for language language models (LLMs). While recent scaling laws have sought to understand how LLMs scale with respect to model size, pre-training data, and computational resources, we focus o…
Model CompressionRingMoE: Mixture-of-Modality-Experts Multi-Modal Foundation Models for Universal Remote Sensing Image Interpretation
The rapid advancement of foundation models has revolutionized visual representation learning in a self-supervised manner. However, their application in remote sensing (RS) remains constrained by a fundamental gap: existi…
Change DetectionDepth EstimationEarth ObservationMixture-of-Experts+3Compositionality Unlocks Deep Interpretable Models
We propose $\chi$-net, an intrinsically interpretable architecture combining the compositional multilinear structure of tensor networks with the expressivity and efficiency of deep neural networks. $\chi$-nets retain equ…
Model CompressionTensor NetworksRandom Conditioning with Distillation for Data-Efficient Diffusion Model Compression
Diffusion models generate high-quality images through progressive denoising but are computationally intensive due to large model sizes and repeated sampling. Knowledge distillation, which transfers knowledge from a compl…
DenoisingKnowledge DistillationModel CompressionPenrose Tiled Low-Rank Compression and Section-Wise Q&A Fine-Tuning: A General Framework for Domain-Specific Large Language Model Adaptation
Large language models (LLMs) hold great promise for specialized scientific domains such as materials science, yet adapting them efficiently and accurately to domain-specific knowledge remains challenging due to limited d…
Language ModelingLanguage ModellingLarge Language ModelLow-rank compression+1Multi-Task Semantic Communications via Large Models
Artificial intelligence (AI) promises to revolutionize the design, optimization and management of next-generation communication systems. In this article, we explore the integration of large AI models (LAMs) into semantic…
Model CompressionRetrieval-augmented GenerationA Low-Power Streaming Speech Enhancement Accelerator For Edge Devices
Transformer-based speech enhancement models yield impressive results. However, their heterogeneous and complex structure restricts model compression potential, resulting in greater complexity and reduced hardware efficie…
Model CompressionSpeech EnhancementDelving Deep into Semantic Relation Distillation
Knowledge distillation has become a cornerstone technique in deep learning, facilitating the transfer of knowledge from complex models to lightweight counterparts. Traditional distillation approaches focus on transferrin…
Knowledge DistillationModel CompressionRelationSuperpixelsMoQa: Rethinking MoE Quantization with Multi-stage Data-model Distribution Awareness
With the advances in artificial intelligence, Mix-of-Experts (MoE) has become the main form of Large Language Models (LLMs), and its demand for model compression is increasing. Quantization is an effective method that no…
Language ModelingLanguage ModellingModel CompressionQuantizationQ-MambaIR: Accurate Quantized Mamba for Efficient Image Restoration
State-Space Models (SSMs) have attracted considerable attention in Image Restoration (IR) due to their ability to scale linearly sequence length while effectively capturing long-distance dependencies. However, deploying …
Computational EfficiencyImage RestorationMambaModel Compression+2Boosting Large Language Models with Mask Fine-Tuning
The model is usually kept integral in the mainstream large language model (LLM) fine-tuning protocols. No works have questioned whether maintaining the integrity of the model is indispensable for performance. In this wor…
Language ModelingLanguage ModellingLarge Language ModelModel Compression+1Temporal Action Detection Model Compression by Progressive Block Drop
Temporal action detection (TAD) aims to identify and localize action instances in untrimmed videos, which is essential for various video understanding tasks. However, recent improvements in model performance, driven by l…
Action DetectionAutonomous DrivingGPUModel Compression+1