Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic Hashing
Unsupervised semantic hashing has emerged as an indispensable technique for fast image search, which aims to convert images into binary hash codes without relying on labels. Recent advancements in the field demonstrate t…
Image RetrievalKnowledge DistillationModel CompressionTowards efficient deep autoencoders for multivariate time series anomaly detection
Multivariate time series anomaly detection is a crucial problem in many industrial and research applications. Timely detection of anomalies allows, for instance, to prevent defects in manufacturing processes and failures…
Anomaly DetectionModel CompressionQuantizationTime Series+1DyCE: Dynamically Configurable Exiting for Deep Learning Compression and Real-time Scaling
Conventional deep learning (DL) model compression and scaling methods focus on altering the model's components, impacting the results across all samples uniformly. However, since samples vary in difficulty, a dynamic mod…
image-classificationImage ClassificationModel CompressionDifferentially Private Knowledge Distillation via Synthetic Text Generation
Large Language models (LLMs) are achieving state-of-the-art performance in many different downstream tasks. However, the increasing urgency of data privacy puts pressure on practitioners to train LLMs with Differential P…
Knowledge DistillationModel CompressionPrivacy PreservingText Generation"Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach
Modern deep neural networks (DNNs) are extremely powerful; however, this comes at the price of increased depth and having more parameters per layer, making their training and inference more computationally challenging. I…
Model CompressionQuantizationPromptMM: Multi-Modal Knowledge Distillation for Recommendation with Prompt-Tuning
Multimedia online platforms (e.g., Amazon, TikTok) have greatly benefited from the incorporation of multimedia (e.g., visual, textual, and acoustic) content into their personal recommender systems. These modalities provi…
Knowledge DistillationModel CompressionRecommendation SystemsLLM Inference Unveiled: Survey and Roofline Model Insights
The field of efficient Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Although the field has expanded and is vibrant, there hasn't been a concise fram…
Knowledge DistillationLanguage ModellingLarge Language Modelmodel+3Model Compression Method for S4 with Diagonal State Space Layers using Balanced Truncation
To implement deep learning models on edge devices, model compression methods have been widely recognized as useful. However, it remains unclear which model compression methods are effective for Structured State Space Seq…
Model CompressionFinGPT-HPC: Efficient Pretraining and Finetuning Large Language Models for Financial Applications with High-Performance Computing
Large language models (LLMs) are computationally intensive. The computation workload and the memory footprint grow quadratically with the dimension (layer width). Most of LLMs' parameters come from the linear layers of t…
GPUModel CompressionQuantizationPromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning
Recent advancements in large language models (LLMs) have raised concerns about inference costs, increasing the need for research into model compression. While knowledge distillation (KD) is a prominent method for this, r…
Instruction FollowingKnowledge DistillationModel CompressionFrom Cloud to Edge: Rethinking Generative AI for Low-Resource Design Challenges
Generative Artificial Intelligence (AI) has shown tremendous prospects in all aspects of technology, including design. However, due to its heavy demand on resources, it is usually trained on large computing infrastructur…
Edge-computingModel CompressionA Survey on Knowledge Distillation of Large Language Models
In the era of Large Language Models (LLMs), Knowledge Distillation (KD) emerges as a pivotal methodology for transferring advanced capabilities from leading proprietary LLMs, such as GPT-4, to their open-source counterpa…
Data AugmentationKnowledge DistillationModel CompressionSurveyTowards a tailored mixed-precision sub-8-bit quantization scheme for Gated Recurrent Units using Genetic Algorithms
Despite the recent advances in model compression techniques for deep neural networks, deploying such models on ultra-low-power embedded devices still proves challenging. In particular, quantization schemes for Gated Recu…
Model CompressionQuantizationExtraction of nonlinearity in neural networks with Koopman operator
Nonlinearity plays a crucial role in deep neural networks. In this paper, we investigate the degree to which the nonlinearity of the neural network is essential. For this purpose, we employ the Koopman operator, extended…
Model CompressionFast Vocabulary Transfer for Language Model Compression
Real-world business applications require a trade-off between language model performance and size. We propose a new method for model compression that relies on vocabulary transfer. We evaluate the method on various vertic…
Language ModelingLanguage ModellingmodelModel CompressionModel Compression and Efficient Inference for Large Language Models: A Survey
Transformer based large language models have achieved tremendous success. However, the significant memory and computational costs incurred during the inference process make it challenging to deploy large models on resour…
Knowledge DistillationModel CompressionQuantizationBayesian Deep Learning Via Expectation Maximization and Turbo Deep Approximate Message Passing
Efficient learning and model compression algorithm for deep neural network (DNN) is a key workhorse behind the rise of deep learning (DL). In this work, we propose a message passing based Bayesian deep learning algorithm…
Bayesian InferenceFederated LearningHandwriting RecognitionModel Compression+1Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy
As Vision Transformers (ViTs) increasingly set new benchmarks in computer vision, their practical deployment on inference engines is often hindered by their significant memory bandwidth and (on-chip) memory footprint req…
Model CompressionL4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models
Due to the high memory and computational costs associated with large language models (LLMs), model compression techniques such as quantization, which reduces inference costs, and parameter-efficient fine-tuning (PEFT) me…
Few-Shot LearningIn-Context LearningModel Compressionparameter-efficient fine-tuning+1QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning
The practical deployment of diffusion models still suffers from the high memory and time overhead. While quantization paves a way for compression and acceleration, existing methods unfortunately fail when the models are …
Image GenerationModel CompressionQuantization