Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
Automated Inference of Graph Transformation Rules
The explosion of data available in life sciences is fueling an increasing demand for expressive models and computational methods. Graph transformation is a model for dynamic systems with a large variety of applications. …
Model CompressionImprove Knowledge Distillation via Label Revision and Data Selection
Knowledge distillation (KD) has become a widely used technique in the field of model compression, which aims to transfer knowledge from a large teacher model to a lightweight student model for efficient network developme…
Knowledge DistillationModel CompressionEnhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations
Large Language Models are growing in size, and we expect them to continue to do so, as larger models train quicker. However, this increase in size will severely impact inference costs. Therefore model compression is impo…
Model CompressionInstance-Aware Group Quantization for Vision Transformers
Post-training quantization (PTQ) is an efficient model compression technique that quantizes a pretrained full-precision model using only a small calibration set of unlabeled samples without retraining. PTQ methods for co…
image-classificationImage ClassificationInstance SegmentationModel Compression+4Streamlining Redundant Layers to Compress Large Language Models
This paper introduces LLM-Streamline, a pioneer work on layer pruning for large language models (LLMs). It is based on the observation that different layers have varying impacts on hidden states, enabling the identificat…
Model CompressionDense Vision Transformer Compression with Few Samples
Few-shot model compression aims to compress a large model into a more compact one with only a tiny training set (even without labels). Block-level pruning has recently emerged as a leading technique in achieving high acc…
Model CompressionIs Modularity Transferable? A Case Study through the Lens of Knowledge Distillation
The rise of Modular Deep Learning showcases its potential in various Natural Language Processing applications. Parameter-efficient fine-tuning (PEFT) modularity has been shown to work for various use cases, from domain a…
Domain AdaptationKnowledge DistillationLanguage ModelingLanguage Modelling+6Tiny Models are the Computational Saver for Large Models
This paper introduces TinySaver, an early-exit-like dynamic model compression approach which employs tiny models to substitute large models adaptively. Distinct from traditional compression techniques, dynamic methods li…
Computational EfficiencyImage ClassificationModel CompressionOrder of Compression: A Systematic and Optimal Sequence to Combinationally Compress CNN
Model compression has gained significant popularity as a means to alleviate the computational and memory demands of machine learning models. Each compression technique leverages unique features to reduce the size of neur…
Knowledge DistillationModel CompressionQuantizationAre Compressed Language Models Less Subgroup Robust?
To reduce the inference cost of large language models, model compression is increasingly used to create smaller scalable models. However, little is known about their robustness to minority subgroups defined by the labels…
Model CompressionMagic for the Age of Quantized DNNs
Recently, the number of parameters in DNNs has explosively increased, as exemplified by LLMs (Large Language Models), making inference on small-scale computers more difficult. Model compression technology is, therefore, …
Model CompressionQuantizationAdvancing IIoT with Over-the-Air Federated Learning: The Role of Iterative Magnitude Pruning
The industrial Internet of Things (IIoT) under Industry 4.0 heralds an era of interconnected smart devices where data-driven insights and machine learning (ML) fuse to revolutionize manufacturing. A noteworthy developmen…
Federated LearningModel CompressionDiPaCo: Distributed Path Composition
Progress in machine learning (ML) has been fueled by scaling neural network models. This scaling has been enabled by ever more heroic feats of engineering, necessary for accommodating ML approaches that require high band…
Language ModellingModel CompressionAdversarial Fine-tuning of Compressed Neural Networks for Joint Improvement of Robustness and Efficiency
As deep learning (DL) models are increasingly being integrated into our everyday lives, ensuring their safety by making them robust against adversarial attacks has become increasingly critical. DL models have been found …
Adversarial RobustnessModel CompressionNetwork PruningQuantizationPYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation
Recently, the scale of transformers has grown rapidly, which introduces considerable challenges in terms of training overhead and inference efficiency in the scope of task adaptation. Existing works, namely Parameter-Eff…
Model Compressionparameter-efficient fine-tuningBRIEDGE: EEG-Adaptive Edge AI for Multi-Brain to Multi-Robot Interaction
Recent advances in EEG-based BCI technologies have revealed the potential of brain-to-robot collaboration through the integration of sensing, computing, communication, and control. In this paper, we present BRIEDGE as an…
EEGModel CompressionQuantizationMaxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons
When training deep neural networks, the phenomenon of $\textit{dying neurons}$ $\unicode{x2013}$units that become inactive or saturated, output zero during training$\unicode{x2013}$ has traditionally been viewed as undes…
Continual LearningModel CompressionSVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression
The advancements in Large Language Models (LLMs) have been hindered by their substantial sizes, which necessitate LLM compression methods for practical deployment. Singular Value Decomposition (SVD) offers a promising so…
Language ModelingLanguage ModellingLarge Language ModelModel CompressionEnhanced Sparsification via Stimulative Training
Sparsification-based pruning has been an important category in model compression. Existing methods commonly set sparsity-inducing penalty terms to suppress the importance of dropped weights, which is regarded as the supp…
Knowledge DistillationModel CompressionOptimal Policy Sparsification and Low Rank Decomposition for Deep Reinforcement Learning
Deep reinforcement learning(DRL) has shown significant promise in a wide range of applications including computer games and robotics. Yet, training DRL policies consume extraordinary computing resources resulting in dens…
Deep Reinforcement LearningEdge-computingModel Compression