Papers Model Compression
“Model Compression” 태그가 달린 논문 1,632편 · 필터 해제
Understanding the Effect of Model Compression on Social Bias in Large Language Models
Large Language Models (LLMs) trained with self-supervision on vast corpora of web text fit to the social biases of that text. Without intervention, these social biases persist in the model's predictions in downstream tas…
Knowledge DistillationModel CompressionQuantizationLanguage Model Knowledge Distillation for Efficient Question Answering in Spanish
Recent advances in the development of pre-trained Spanish language models has led to significant progress in many Natural Language Processing (NLP) tasks, such as question answering. However, the lack of efficient models…
Knowledge DistillationLanguage ModelingLanguage ModellingModel Compression+1The Efficiency Spectrum of Large Language Models: An Algorithmic Survey
The rapid growth of Large Language Models (LLMs) has been a driving force in transforming various domains, reshaping the artificial general intelligence landscape. However, the increasing computational and memory demands…
Model CompressionSurveyPhysics Inspired Criterion for Pruning-Quantization Joint Learning
Pruning-quantization joint learning always facilitates the deployment of deep neural networks (DNNs) on resource-constrained edge devices. However, most existing methods do not jointly learn a global criterion for prunin…
image-classificationImage ClassificationModel CompressionQuantizationPrivacy and Accuracy Implications of Model Complexity and Integration in Heterogeneous Federated Learning
Federated Learning (FL) has been proposed as a privacy-preserving solution for distributed machine learning, particularly in heterogeneous FL settings where clients have varying computational capabilities and thus train …
Federated LearningInference AttackMembership Inference AttackModel Compression+1Towards Higher Ranks via Adversarial Weight Pruning
Convolutional Neural Networks (CNNs) are hard to deploy on edge devices due to its high computation and storage complexities. As a common practice for model compression, network pruning consists of two major categories: …
Model CompressionNetwork PruningLayerCollapse: Adaptive compression of neural networks
Handling the ever-increasing scale of contemporary deep learning and transformer-based models poses a significant challenge. Overparameterized Transformer networks outperform prior art in Natural Language processing and …
Computational Efficiencyimage-classificationImage ClassificationKnowledge Distillation+3Relationship between Model Compression and Adversarial Robustness: A Review of Current Evidence
Increasing the model capacity is a known approach to enhance the adversarial robustness of deep learning networks. On the other hand, various model compression techniques, including pruning and quantization, can reduce t…
Adversarial RobustnessModel CompressionQuantizationCosine Similarity Knowledge Distillation for Individual Class Information Transfer
Previous logits-based Knowledge Distillation (KD) have utilized predictions about multiple categories within each sample (i.e., class predictions) and have employed Kullback-Leibler (KL) divergence to reduce the discrepa…
Knowledge DistillationModel CompressionEducation distillation:getting student models to learn in shcools
Knowledge distillation is one of the methods for model compression, and existing knowledge distillation techniques focus on how to improve the distillation algorithm so as to enhance the distillation efficiency. This pap…
Incremental LearningKnowledge DistillationModel CompressionKnowledge Distillation Based Semantic Communications For Multiple Users
Deep learning (DL) has shown great potential in revolutionizing the traditional communications system. Many applications in communications have adopted DL techniques due to their powerful representation ability. However,…
DecoderKnowledge DistillationModel CompressionSemantic Communication+1Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper
This paper delves into the pressing need in Parameter-Efficient Fine-Tuning (PEFT) for Large Language Models (LLMs). While LLMs possess remarkable capabilities, their extensive parameter requirements and associated compu…
Model Compressionparameter-efficient fine-tuningPositionCompact 3D Gaussian Representation for Radiance Field
Neural Radiance Fields (NeRFs) have demonstrated remarkable potential in capturing complex 3D scenes with high fidelity. However, one persistent challenge that hinders the widespread adoption of NeRFs is the computationa…
3DGSModel CompressionNovel View SynthesisQuantizationEfficient Transformer Knowledge Distillation: A Performance Review
As pretrained transformer language models continue to achieve state-of-the-art performance, the Natural Language Processing community has pushed for advances in model compression and efficient attention mechanisms to add…
Knowledge DistillationModel Compressionnamed-entity-recognitionNamed Entity Recognition+2Shedding the Bits: Pushing the Boundaries of Quantization with Minifloats on FPGAs
Post-training quantization (PTQ) is a powerful technique for model compression, reducing the numerical precision in neural networks without additional training overhead. Recent works have investigated adopting 8-bit floa…
Model CompressionQuantizationLQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning
We propose a simple approach for memory-efficient adaptation of pretrained language models. Our approach uses an iterative algorithm to decompose each pretrained matrix into a high-precision low-rank component and a memo…
GPULanguage ModelingLanguage ModellingModel Compression+1Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review
The field of Tiny Machine Learning (TinyML) has gained significant attention due to its potential to enable intelligent applications on resource-constrained devices. This review provides an in-depth analysis of the advan…
Model CompressionQuantizationA Speed Odyssey for Deployable Quantization of LLMs
The large language model era urges faster and less costly inference. Prior model compression works on LLMs tend to undertake a software-centric approach primarily focused on the simulated quantization performance. By neg…
Language ModelingLanguage ModellingLarge Language ModelModel Compression+1On the Impact of Calibration Data in Post-training Quantization and Pruning
Quantization and pruning form the foundation of compression for neural networks, enabling efficient inference for large language models (LLMs). Recently, various quantization and pruning techniques have demonstrated rema…
Model CompressionQuantizationFedCode: Communication-Efficient Federated Learning via Transferring Codebooks
Federated Learning (FL) is a distributed machine learning paradigm that enables learning models from decentralized local data. While FL offers appealing properties for clients' data privacy, it imposes high communication…
Federated LearningModel Compression