Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers
Recent advancements in diffusion models, particularly the architectural transformation from UNet-based models to Diffusion Transformers (DiTs), significantly improve the quality and scalability of image and video generat…
Image GenerationModel CompressionQuantizationVideo GenerationPruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging
While large language models (LLMs) excel in many domains, their complexity and scale challenge deployment in resource-limited environments. Current compression techniques, such as parameter pruning, often fail to effecti…
MMLUModel CompressionQuantizationSpeeding Up Image Classifiers with Little Companions
Scaling up neural networks has been a key recipe to the success of large language and vision models. However, in practice, up-scaled models can be disproportionately costly in terms of computations, providing only margin…
image-classificationImage ClassificationModel CompressionExploring compressibility of transformer based text-to-music (TTM) models
State-of-the art Text-To-Music (TTM) generative AI models are large and require desktop or server class compute, making them infeasible for deployment on mobile phones. This paper presents an analysis of trade-offs betwe…
DecoderFADKnowledge DistillationModel CompressionLiteYOLO-ID: A Lightweight Object Detection Network for Insulator Defect Detection
Insulator defect detection is of great significance to ensure the normal operation of power transmission and distribution networks. In response to the problems of low speed, low accuracy, and difficulty in deploying to e…
Defect DetectionInsulator Defect DetectionModel Compressionobject-detection+1MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression
Sparse attention can effectively mitigate the significant memory and throughput demands of Large Language Models (LLMs) in long contexts. Existing methods typically employ a uniform sparse attention mask, applying the sa…
GPULanguage ModelingLanguage ModellingLarge Language Model+2Reinforced Knowledge Distillation for Time Series Regression
As one of the most popular and effective methods in model compression, knowledge distillation (KD) attempts to transfer knowledge from single or multiple large-scale networks (i.e., Teachers) to a compact network (i.e., …
Knowledge DistillationModel CompressionregressionTime Series+1Failure-Resilient Distributed Inference with Model Compression over Heterogeneous Edge Devices
The distributed inference paradigm enables the computation workload to be distributed across multiple devices, facilitating the implementations of deep learning based intelligent services on extremely resource-constraine…
Knowledge DistillationModel CompressionFLoCoRA: Federated learning compression with low-rank adaptation
Low-Rank Adaptation (LoRA) methods have gained popularity in efficient parameter fine-tuning of models containing hundreds of billions of parameters. In this work, instead, we demonstrate the application of LoRA methods …
Federated LearningModel CompressionQuantizationSDQ: Sparse Decomposed Quantization for LLM Inference
Recently, large language models (LLMs) have shown surprising performance in task-specific workloads as well as general tasks with the given prompts. However, to achieve unprecedented performance, recent LLMs use billions…
Model CompressionQuantizationFinding Task-specific Subnetworks in Multi-task Spoken Language Understanding Model
Recently, multi-task spoken language understanding (SLU) models have emerged, designed to address various speech processing tasks. However, these models often rely on a large number of parameters. Also, they often encoun…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Emotion Recognitionintent-classification+6Model Adaptation for Time Constrained Embodied Control
When adopting a deep learning model for embodied agents, it is required that the model structure be optimized for specific tasks and operational conditions. Such optimization can be static such as model compression or dy…
Autonomous DrivingDecision MakingmodelModel Compression+1An Empirical Investigation of Matrix Factorization Methods for Pre-trained Transformers
The increasing size of transformer-based models in NLP makes the question of compressing them important. In this work, we present a comprehensive analysis of factorization based model compression techniques. Specifically…
Model Compressiontext-classificationText ClassificationCompress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
Fine-tuning large language models (LLMs) with low-rank adaptations (LoRAs) has become common practice, often yielding numerous copies of the same LLM differing only in their LoRA updates. This paradigm presents challenge…
GPUModel CompressionKnowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions
Federated Learning (FL) is a distributed and privacy-preserving machine learning paradigm that coordinates multiple clients to train a model while keeping the raw data localized. However, this traditional FL poses some c…
Federated LearningKnowledge DistillationModel CompressionPrivacy Preserving+1Implicit Neural Representation for Videos Based on Residual Connection
Video compression technology is essential for transmitting and storing videos. Many video compression methods reduce information in videos by removing high-frequency components and utilizing similarities between frames. …
Image ReconstructionModel CompressionVideo CompressionPC-LoRA: Low-Rank Adaptation for Progressive Model Compression with Knowledge Distillation
Low-rank adaption (LoRA) is a prominent method that adds a small number of learnable parameters to the frozen pre-trained weights for parameter-efficient fine-tuning. Prompted by the question, ``Can we make its represent…
Knowledge DistillationModel Compressionparameter-efficient fine-tuningEncCluster: Scalable Functional Encryption in Federated Learning through Weight Clustering and Probabilistic Filters
Federated Learning (FL) enables model training across decentralized devices by communicating solely local model updates to an aggregation server. Although such limited data sharing makes FL more secure than centralized a…
Federated LearningModel CompressionDistilDoc: Knowledge Distillation for Visually-Rich Document Applications
This work explores knowledge distillation (KD) for visually-rich document (VRD) applications such as document layout analysis (DLA) and document image classification (DIC). While VRD research is dependent on increasingly…
document-image-classificationDocument Image ClassificationDocument Layout Analysisdocument understanding+6Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark
Large Language Models~(LLMs) have become foundational in the realm of natural language processing, demonstrating performance improvements as model sizes increase. The Mixture-of-Experts~(MoE) approach offers a promising …
BenchmarkingMixture-of-ExpertsModel CompressionQuantization