Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
Artemis: HE-Aware Training for Efficient Privacy-Preserving Machine Learning
Privacy-Preserving ML (PPML) based on Homomorphic Encryption (HE) is a promising foundational privacy technology. Making it more practical requires lowering its computational cost, especially, in handling modern large de…
Model CompressionPrivacy PreservingDistilling Inductive Bias: Knowledge Distillation Beyond Model Compression
With the rapid development of computer vision, Vision Transformers (ViTs) offer the tantalizing prospect of unified information processing across visual and textual domains. But due to the lack of inherent inductive bias…
Inductive BiasKnowledge DistillationmodelModel CompressionBridging the Gap Between Foundation Models and Heterogeneous Federated Learning
Federated learning (FL) offers privacy-preserving decentralized machine learning, optimizing models at edge clients without sharing private data. Simultaneously, foundation models (FMs) have gained traction in the artifi…
Federated LearningModel CompressionPrivacy PreservingCAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs
Vision Transformers (ViTs) have emerged as state-of-the-art models for various vision tasks recently. However, their heavy computation costs remain daunting for resource-limited devices. Consequently, researchers have de…
Model CompressionSemantic SegmentationOn the Impact of Quantization and Pruning of Self-Supervised Speech Models for Downstream Speech Recognition Tasks "In-the-Wild''
Recent advances with self-supervised learning have allowed speech recognition systems to achieve state-of-the-art (SOTA) word error rates (WER) while requiring only a fraction of the labeled training data needed by its p…
Data AugmentationModel CompressionQuantizationSelf-Supervised Learning+2VIC-KD: Variance-Invariance-Covariance Knowledge Distillation to Make Keyword Spotting More Robust Against Adversarial Attacks
Keyword spotting (KWS) refers to the task of identifying a set of predefined words in audio streams. With the advances seen recently with deep neural networks, it has become a popular technology to activate and control s…
Adversarial RobustnessKeyword SpottingKnowledge DistillationModel CompressionPruning Large Language Models via Accuracy Predictor
Large language models(LLMs) containing tens of billions of parameters (or even more) have demonstrated impressive capabilities in various NLP tasks. However, substantial model size poses challenges to training, inference…
MMLUModel CompressionTraining dynamic models using early exits for automatic speech recognition on resource-constrained devices
The ability to dynamically adjust the computational load of neural models during inference is crucial for on-device processing scenarios characterised by limited and time-varying computational resources. A promising solu…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Model Compressionspeech-recognition+1Two-Step Knowledge Distillation for Tiny Speech Enhancement
Tiny, causal models are crucial for embedded audio machine learning applications. Model compression can be achieved via distilling knowledge from a large teacher into a smaller student model. In this work, we propose a n…
Knowledge DistillationModel CompressionSpeech EnhancementCoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders
Large-scale self-supervised pre-trained speech encoders outperform conventional approaches in speech recognition and translation tasks. Due to the high cost of developing these large models, building new encoders for new…
Contrastive LearningKnowledge DistillationModel Compressionspeech-recognition+4Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantization
Low Rank Decomposition (LRD) is a model compression technique applied to the weight tensors of deep learning models in order to reduce the number of trainable parameters and computational complexity. However, due to high…
Model CompressionQuantizationNorm Tweaking: High-performance Low-bit Quantization of Large Language Models
As the size of large language models (LLMs) continues to grow, model compression without sacrificing accuracy has become a crucial challenge for deployment. While some quantization methods, such as GPTQ, have made progre…
Model CompressionQuantizationCompressing Vision Transformers for Low-Resource Visual Learning
Vision transformer (ViT) and its variants have swept through visual learning leaderboards and offer state-of-the-art accuracy in tasks such as image classification, object detection, and semantic segmentation by attendin…
Autonomous Navigationimage-classificationImage ClassificationModel Compression+4ADC/DAC-Free Analog Acceleration of Deep Neural Networks with Frequency Transformation
The edge processing of deep neural networks (DNNs) is becoming increasingly important due to its ability to extract valuable information directly at the data source to minimize latency and energy consumption. Frequency-d…
Computational EfficiencyModel CompressionUncovering the Hidden Cost of Model Compression
In an age dominated by resource-intensive foundation models, the ability to efficiently adapt to downstream tasks is crucial. Visual Prompting (VP), drawing inspiration from the prompting techniques employed in Large Lan…
modelModel CompressionQuantizationTransfer Learning+1Computation-efficient Deep Learning for Computer Vision: A Survey
Over the past decade, deep learning models have exhibited considerable advancements, reaching or even exceeding human-level performance in a range of visual perception tasks. This remarkable progress has sparked interest…
Autonomous VehiclesDeep LearningEdge-computingModel Compression+1Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning
The use of large transformer-based models such as BERT, GPT, and T5 has led to significant advancements in natural language processing. However, these models are computationally expensive, necessitating model compression…
Knowledge DistillationModel CompressionOmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models
Large language models (LLMs) have revolutionized natural language processing tasks. However, their practical deployment is hindered by their immense memory and computation requirements. Although recent post-training quan…
Common Sense ReasoningComputational EfficiencyGPULarge Language Model+2DLIP: Distilling Language-Image Pre-training
Vision-Language Pre-training (VLP) shows remarkable progress with the assistance of extremely heavy parameters, which challenges deployment in real applications. Knowledge distillation is well recognized as the essential…
Image CaptioningImage-text RetrievalKnowledge DistillationModel Compression+4QD-BEV : Quantization-aware View-guided Distillation for Multi-view 3D Object Detection
Multi-view 3D detection based on BEV (bird-eye-view) has recently achieved significant improvements. However, the huge memory consumption of state-of-the-art models makes it hard to deploy them on vehicles, and the non-t…
3D Object DetectionModel Compressionobject-detectionObject Detection+1