Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?
Motivated by reducing the computational and storage costs of LLMs, model compression and KV cache compression have attracted much attention from researchers. However, current methods predominantly emphasize maintaining t…
Arithmetic ReasoningCommon Sense ReasoningModel CompressionRetrieval-augmented GenerationSwallowing the Poison Pills: Insights from Vulnerability Disparity Among LLMs
Modern large language models (LLMs) exhibit critical vulnerabilities to poison pill attacks: localized data poisoning that alters specific factual knowledge while preserving overall model utility. We systematically demon…
Data PoisoningDiagnosticMemorizationMMLU+2When Compression Meets Model Compression: Memory-Efficient Double Compression for Large Language Models
Large language models (LLMs) exhibit excellent performance in various tasks. However, the memory requirements of LLMs present a great challenge when deploying on memory-limited devices, even for quantized LLMs. This pape…
Model CompressionQuantizationVision Foundation Models in Medical Image Analysis: Advances and Challenges
The rapid development of Vision Foundation Models (VFMs), particularly Vision Transformers (ViT) and Segment Anything Model (SAM), has sparked significant advances in the field of medical image analysis. These models hav…
Domain AdaptationFederated LearningImage SegmentationKnowledge Distillation+5Efficient AI in Practice: Training and Deployment of Efficient LLMs for Industry Applications
Large language models (LLMs) have demonstrated remarkable performance across a wide range of industrial applications, from search and recommendations to generative tasks. Although scaling laws indicate that larger models…
Knowledge DistillationModel CompressionQuantizationOptimizing Singular Spectrum for Large Language Model Compression
Large language models (LLMs) have demonstrated remarkable capabilities, yet prohibitive parameter complexity often hinders their deployment. Existing singular value decomposition (SVD) based compression methods simply de…
Language ModelingLanguage ModellingLarge Language Modelmodel+1MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures
The remarkable performance of large language models (LLMs) in various language tasks has attracted considerable attention. However, the ever-increasing size of these models presents growing challenges for deployment and …
Model CompressionEvery Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
With the emergence of Mixture-of-Experts (MoE), the efficient scaling of model size has accelerated the development of large language models in recent years. However, their high memory requirements prevent their use in r…
Knowledge DistillationMixture-of-ExpertsModel CompressionEnhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping
Knowledge Distillation (KD) has emerged as a prominent technique for model compression. However, conventional KD approaches primarily focus on homogeneous architectures with identical tokenizers, constraining their appli…
Code GenerationInstruction FollowingKnowledge DistillationMath+1OPTISHEAR: Towards Efficient and Adaptive Pruning of Large Language Models via Evolutionary Optimization
Post-training pruning has emerged as a crucial optimization technique as large language models (LLMs) continue to grow rapidly. However, the significant variations in weight distributions across different LLMs make fixed…
Model CompressionForget the Data and Fine-Tuning! Just Fold the Network to Compress
We introduce model folding, a novel data-free model compression technique that merges structurally similar neurons across layers, significantly reducing the model size without the need for fine-tuning or access to traini…
Model CompressionDarwinLM: Evolutionary Structured Pruning of Large Language Models
Large Language Models (LLMs) have achieved significant success across various NLP tasks. However, their massive computational costs limit their widespread use, particularly in real-time applications. Structured pruning o…
Model CompressionVision-Language Models for Edge Networks: A Comprehensive Survey
Vision Large Language Models (VLMs) combine visual understanding with natural language processing, enabling tasks like image captioning, visual question answering, and video analysis. While VLMs show impressive capabilit…
Autonomous VehiclesImage CaptioningKnowledge DistillationModel Compression+4Runtime Tunable Tsetlin Machines for Edge Inference on eFPGAs
Embedded Field-Programmable Gate Arrays (eFPGAs) allow for the design of hardware accelerators of edge Machine Learning (ML) applications at a lower power budget compared with traditional FPGA platforms. However, the lim…
Model CompressionResynthesisLow-Rank Compression for IMC Arrays
In this study, we address the challenge of low-rank model compression in the context of in-memory computing (IMC) architectures. Traditional pruning approaches, while effective in model size reduction, necessitate additi…
Low-rank compressionModel CompressionSystematic Outliers in Large Language Models
Outliers have been widely observed in Large Language Models (LLMs), significantly impacting model performance and posing challenges for model compression. Understanding the functionality and formation mechanisms of these…
Model CompressionSynergistic Effects of Knowledge Distillation and Structured Pruning for Self-Supervised Speech Models
Traditionally, Knowledge Distillation (KD) is used for model compression, often leading to suboptimal performance. In this paper, we evaluate the impact of combining KD loss with alternative pruning techniques, including…
Knowledge DistillationModel CompressionSelf-Supervised LearningActivation-Informed Merging of Large Language Models
Model merging, a method that combines the parameters and embeddings of multiple fine-tuned large language models (LLMs), offers a promising approach to enhance model performance across various tasks while maintaining com…
Computational EfficiencyContinual LearningModel CompressionTheoretical Guarantees for Low-Rank Compression of Deep Neural Networks
Deep neural networks have achieved state-of-the-art performance across numerous applications, but their high memory and computational demands present significant challenges, particularly in resource-constrained environme…
Low-rank compressionModel CompressionMIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Tasks
Multimodal fusion leverages information across modalities to learn better feature representations with the goal of improving performance in fusion-based tasks. However, multimodal datasets, especially in medical settings…
ImputationKnowledge DistillationModel Compression