Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity
Linear recurrent neural networks enable powerful long-range sequence modeling with constant memory usage and time-per-token during inference. These architectures hold promise for streaming applications at the edge, but d…
Audio DenoisingDenoisingGPUModel CompressionHuff-LLM: End-to-End Lossless Compression for Efficient LLM Inference
As they become more capable, large language models (LLMs) have continued to rapidly increase in size. This has exacerbated the difficulty in running state of the art LLMs on small, edge devices. Standard techniques advoc…
Model CompressionQuantizationRole of Mixup in Topological Persistence Based Knowledge Distillation for Wearable Sensor Data
The analysis of wearable sensor data has enabled many successes in several applications. To represent the high-sampling rate time-series with sufficient detail, the use of topological data analysis (TDA) has been conside…
Data AugmentationKnowledge DistillationModel CompressionTime Series+2Attention Sinks and Outlier Features: A 'Catch, Tag, and Release' Mechanism for Embeddings
Two prominent features of large language models (LLMs) is the presence of large-norm (outlier) features and the tendency for tokens to attend very strongly to a select few tokens. Despite often having no semantic relevan…
Model CompressionTAGPivoting Factorization: A Compact Meta Low-Rank Representation of Sparsity for Efficient Inference in Large Language Models
The rapid growth of Large Language Models has driven demand for effective model compression techniques to reduce memory and computation costs. Low-rank pruning has gained attention for its tensor coherence and GPU compat…
GPUModel CompressionEfficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compression
ASR systems are deployed across diverse environments, each with specific hardware constraints. We use supernet training to jointly train multiple encoders of varying sizes, enabling dynamic model size adjustment to fit h…
Model CompressionSANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer
This paper presents SANA-1.5, a linear Diffusion Transformer for efficient scaling in text-to-image generation. Building upon SANA-1.0, we introduce three key innovations: (1) Efficient Training Scaling: A depth-growth p…
Image GenerationModel CompressionText to Image GenerationText-to-Image GenerationPerforated Backpropagation: A Neuroscience Inspired Extension to Artificial Neural Networks
The neurons of artificial neural networks were originally invented when much less was known about biological neurons than is known today. Our work explores a modification to the core neuron unit to make it more parallel …
Drug DiscoveryLanguage ModelingModel CompressionStock Trend PredictionTAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
Causal language models have demonstrated remarkable capabilities, but their size poses significant challenges for deployment in resource-constrained environments. Knowledge distillation, a widely-used technique for trans…
Knowledge DistillationModel CompressionTransfer LearningOn Accelerating Edge AI: Optimizing Resource-Constrained Environments
Resource-constrained edge deployments demand AI solutions that balance high performance with stringent compute, memory, and energy limitations. In this survey, we present a comprehensive overview of the primary strategie…
Knowledge DistillationModel CompressionNeural Architecture SearchQuantization+2You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning
The ever-increasing size of large language models (LLMs) presents significant challenges for deployment due to their heavy computational and memory requirements. Current model pruning techniques attempt to alleviate thes…
Model CompressionSwiftPrune: Hessian-Free Weight Pruning for Large Language Models
Post-training pruning, as one of the key techniques for compressing large language models, plays a vital role in lightweight model deployment and model sparsity. However, current mainstream pruning methods dependent on t…
Model CompressionQuantizationPractical quantum federated learning and its experimental demonstration
Federated learning is essential for decentralized, privacy-preserving model training in the data-driven era. Quantum-enhanced federated learning leverages quantum resources to address privacy and scalability challenges, …
Federated LearningModel CompressionPrivacy PreservingMultiPruner: Balanced Structure Removal in Foundation Models
Recently, state-of-the-art approaches for pruning large pre-trained models (LPMs) have demonstrated that the training-free removal of non-critical residual blocks in Transformers is viable for reducing model size, achiev…
Model CompressionKnowledge Distillation for Image Restoration : Simultaneous Learning from Degraded and Clean Images
Model compression through knowledge distillation has seen extensive application in classification and segmentation tasks. However, its potential in image-to-image translation, particularly in image restoration, remains u…
DecoderImage ReconstructionImage RestorationImage-to-Image Translation+2Atleus: Accelerating Transformers on the Edge Enabled by 3D Heterogeneous Manycore Architectures
Transformer architectures have become the standard neural network model for various machine learning applications including natural language processing and computer vision. However, the compute and memory requirements in…
Model CompressionQuantizationFASP: Fast and Accurate Structured Pruning of Large Language Models
The rapid increase in the size of large language models (LLMs) has significantly escalated their computational and memory demands, posing challenges for efficient deployment, especially on resource-constrained devices. S…
GPUModel CompressionSWSC: Shared Weight for Similar Channel in LLM
Large language models (LLMs) have spurred development in multiple industries. However, the growing number of their parameters brings substantial storage and computing burdens, making it essential to explore model compres…
Model CompressionA Survey on Dynamic Neural Networks: from Computer Vision to Multi-modal Sensor Fusion
Model compression is essential in the deployment of large Computer Vision models on embedded devices. However, static optimization techniques (e.g. pruning, quantization, etc.) neglect the fact that different inputs have…
Dynamic neural networksModel CompressionSensor FusionMerging Feed-Forward Sublayers for Compressed Transformers
With the rise and ubiquity of larger deep learning models, the need for high-quality compression techniques is growing in order to deploy these models widely. The sheer parameter count of these models makes it difficult …
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+2