A Survey on Model Compression for Large Language Models
Large Language Models (LLMs) have transformed natural language processing tasks successfully. Yet, their large size and high computational needs pose challenges for practical use, especially in resource-limited settings. Model compression has emerged as a key research area to address these challenges. This paper presents a survey of model compression techniques for LLMs. We cover methods like quantization, pruning, and knowledge distillation, highlighting recent advancements. We also discuss benchmarking strategies and evaluation metrics crucial for assessing compressed LLMs. This survey offers valuable insights for researchers and practitioners, aiming to enhance efficiency and real-world applicability of LLMs while laying a foundation for future advancements.
Code (0)
등록된 구현이 없습니다.
Tasks
BenchmarkingKnowledge DistillationModel CompressionQuantizationSurveySimilar Papers 제목 키워드 기반
A Comprehensive Survey of Compression Algorithms for Language Models
How can we compress language models without sacrificing accuracy? The number of compression algorithms for language models is rapidly growing to benefit from remarkable advances of recent language models without side eff…
Knowledge DistillationQuantizationSurveyA Survey on Transformer Compression
Transformer plays a vital role in the realms of natural language processing (NLP) and computer vision (CV), specially for constructing large language models (LLM) and large vision models (LVM). Model compression methods …
Knowledge DistillationMambaModel CompressionQuantization+1Prompt Compression for Large Language Models: A Survey
Leveraging large language models (LLMs) for complex natural language tasks typically requires long-form prompts to convey detailed requirements and information, which results in increased memory usage and inference costs…
parameter-efficient fine-tuningSurveyA Survey of Token Compression for Efficient Multimodal Large Language Models
Multimodal large language models (MLLMs) have made remarkable strides, largely driven by their ability to process increasingly long and complex contexts, such as high-resolution images, extended video sequences, and leng…
Connector-S: A Survey of Connectors in Multi-modal Large Language Models
With the rapid advancements in multi-modal large language models (MLLMs), connectors play a pivotal role in bridging diverse modalities and enhancing model performance. However, the design and evolution of connectors hav…
Mixture-of-ExpertsSurvey