Prompt Compression for Large Language Models: A Survey
Leveraging large language models (LLMs) for complex natural language tasks typically requires long-form prompts to convey detailed requirements and information, which results in increased memory usage and inference costs. To mitigate these challenges, multiple efficient methods have been proposed, with prompt compression gaining significant research interest. This survey provides an overview of prompt compression techniques, categorized into hard prompt methods and soft prompt methods. First, the technical approaches of these methods are compared, followed by an exploration of various ways to understand their mechanisms, including the perspectives of attention optimization, Parameter-Efficient Fine-Tuning (PEFT), modality integration, and new synthetic language. We also examine the downstream adaptations of various prompt compression techniques. Finally, the limitations of current prompt compression methods are analyzed, and several future directions are outlined, such as optimizing the compression encoder, combining hard and soft prompts methods, and leveraging insights from multimodality.
Code (2)
Tasks
parameter-efficient fine-tuningSurveyMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Efficient Prompting Methods for Large Language Models: A Survey
Prompting is a mainstream paradigm for adapting large language models to specific natural language processing tasks without modifying internal parameters. Therefore, detailed supplementary knowledge needs to be integrate…
In-Context LearningPrompt EngineeringSurveyA Survey on Model Compression for Large Language Models
Large Language Models (LLMs) have transformed natural language processing tasks successfully. Yet, their large size and high computational needs pose challenges for practical use, especially in resource-limited settings.…
BenchmarkingKnowledge DistillationModel CompressionQuantization+1A Comprehensive Survey of Compression Algorithms for Language Models
How can we compress language models without sacrificing accuracy? The number of compression algorithms for language models is rapidly growing to benefit from remarkable advances of recent language models without side eff…
Knowledge DistillationQuantizationSurveyA Survey on Transformer Compression
Transformer plays a vital role in the realms of natural language processing (NLP) and computer vision (CV), specially for constructing large language models (LLM) and large vision models (LVM). Model compression methods …
Knowledge DistillationMambaModel CompressionQuantization+1Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
We formalize the problem of prompt compression for large language models (LLMs) and present a framework to unify token-level prompt compression methods which create hard prompts for black-box models. We derive the distor…
Natural Language Queries