Density-aware Soft Context Compression with Semi-Dynamic Compression Ratio
Soft context compression reduces the computational workload of processing long contexts in LLMs by encoding long context into a smaller number of latent tokens. However, existing frameworks apply uniform compression ratios, failing to account for the extreme variance in natural language information density. While adopting a density-aware dynamic compression ratio seems intuitive, empirical investigations reveal that models struggle intrinsically with operations parameterized by input dependent, continuous structural hyperparameters. To resolve this pitfall, we introduce Semi-Dynamic Context Compression framework. Our approach features a Discrete Ratio Selector, which predicts a compression target based on intrinsic information density and quantizes it to a predefined set of discrete compression ratios. It is efficiently jointly trained with the compressor on synthetic data, with the summary lengths as a proxy to create labels for compression ratio prediction. Extensive evaluations confirm that our density-aware framework, utilizing mean pooling as the backbone, consistently outperforms static baselines, establishing a robust Pareto frontier for context compression techniques. Our code, data and model weights are available at https://github.com/yuyijiong/semi-dynamic-context-compress
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokens
Large Language Models (LLMs) face computational inefficiencies and redundant processing when handling long context inputs, prompting a focus on compression techniques. While existing semantic vector-based compression met…
Neighbor Does Matter: Density-Aware Contrastive Learning for Medical Semi-supervised Segmentation
In medical image analysis, multi-organ semi-supervised segmentation faces challenges such as insufficient labels and low contrast in soft tissues. To address these issues, existing studies typically employ semi-supervise…
Contrastive LearningImage SegmentationMedical Image AnalysisMedical Image Segmentation+3Rethinking Soft Compression in Retrieval-Augmented Generation: A Query-Conditioned Selector Perspective
Retrieval-Augmented Generation (RAG) effectively grounds Large Language Models (LLMs) with external knowledge and is widely applied to Web-related tasks. However, its scalability is hindered by excessive context length a…
Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors
Recent advancements in large video-language models have revolutionized video understanding tasks. However, their efficiency is significantly constrained by processing high volumes of visual tokens. Existing token compres…
DAC: A Dynamic Attention-aware Approach for Task-Agnostic Prompt Compression
Task-agnostic prompt compression leverages the redundancy in natural language to reduce computational overhead and enhance information density within prompts, especially in long-context scenarios. Existing methods predom…
GSM8K