CCF: A Context Compression Framework for Efficient Long-Sequence Language Modeling

2025-09-11 · Wenhao Li, Bangcheng Sun, Weihao Ye, Tianyi Zhang, Daohai Yu, Fei Chao, Rongrong Ji arxiv

Scaling language models to longer contexts is essential for capturing rich dependencies across extended discourse. However, naïve context extension imposes significant computational and memory burdens, often resulting in inefficiencies during both training and inference. In this work, we propose CCF, a novel context compression framework designed to enable efficient long-context modeling by learning hierarchical latent representations that preserve global semantics while aggressively reducing input redundancy. CCF integrates segment-wise semantic aggregation with key-value memory encoding, forming compact representations that support accurate reconstruction and long-range understanding. To further enhance scalability, we introduce a training-efficient optimization strategy that couples incremental segment decoding with sparse reservoir sampling, substantially reducing memory overhead without degrading performance. Empirical results on multiple long-context language modeling benchmarks demonstrate that CCF achieves competitive perplexity under high compression ratios, and significantly improves throughput and memory efficiency compared to existing approaches. These findings highlight the potential of structured compression for scalable and effective long-context language modeling.

📄 PDF Abstract BibTeX arXiv:2509.09199

Code (0)

등록된 구현이 없습니다.

UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compression

2025-09-19 · Chenlong Deng, Zhisong Zhang, Kelong Mao, Shuaiyi Li 외 arxiv

Large language models are increasingly capable of handling long-context inputs, but the memory overhead of key-value (KV) cache remains a major bottleneck for general-purpose deployment. While various compression strateg…

Sentence-Anchored Gist Compression for Long-Context LLMs

2025-11-11 · Dmitrii Tarasov, Elizaveta Goncharova, Kuznetsov Andrey arxiv

This work investigates context compression for Large Language Models (LLMs) using learned compression tokens to reduce the memory and computational demands of processing long sequences. We demonstrate that pre-trained LL…

Can Vision-Language Models Handle Long-Context Code? An Empirical Study on Visual Compression

2026-01-31 · Jianping Zhong, Guochang Li, Chen Zhi, Junxiao Han 외 arxiv

Large Language Models (LLMs) struggle with long-context code due to window limitations. Existing textual code compression methods mitigate this via selective filtering but often disrupt dependency closure, causing semant…

Question Answering Code Completion

Glyph: Scaling Context Windows via Visual-Text Compression

2025-10-20 · Jiale Cheng, Yusen Liu, Xinyu Zhang, Yulin Fei 외 arxiv

Large language models (LLMs) increasingly rely on long-context modeling for tasks such as document understanding, code analysis, and multi-step reasoning. However, scaling context windows to the million-token level bring…

End-to-End Context Compression at Scale

2026-06-08 · Ang Li, Sean McLeish, Haozhe Chen, Nimit Kalra 외 arxiv

Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or requir…