No Mean Feat: Simple, Strong Baselines for Context Compression
Context compression reduces Transformer inference costs by replacing lengthy inputs with shorter pre-computed representations. It carries significant benefits for retrieval-augmented generation (RAG) and has attracted growing research attention. However, progress remains difficult to measure due to inconsistent evaluations and baselines. We design a standard, easy-to-reproduce evaluation suite for context compression, BenchPress, along with simple, high-performance baselines for English reading comprehension. BenchPress supports benchmarking across model scales, datasets, compression ratios, and short ($<$1K tokens) to mid-range ($<$8K tokens) contexts. While the suite is applicable to any compression paradigm, our baselines target soft context compression. We establish two simple baselines that strongly outperform the widely used causal compression-token approach: mean pooling and a bidirectional compression-token variant. Our results show the benefit of bidirectional attention when computing compressed representations, and that simple pooling is an expressive compression operator.
Code (0)
등록된 구현이 없습니다.
Tasks
Reading ComprehensionSimilar Papers 제목 키워드 기반
Simplification Using Paraphrases and Context-Based Lexical Substitution
Lexical simplification involves identifying complex words or phrases that need to be simplified, and recommending simpler meaning-preserving substitutes that can be more easily understood. We propose a complex word ident…
Complex Word IdentificationLexical SimplificationText SimplificationContext-aware Neural Machine Translation with Mini-batch Embedding
It is crucial to provide an inter-sentence context in Neural Machine Translation (NMT) models for higher-quality translation. With the aim of using a simple approach to incorporate inter-sentence information, we propose …
Machine TranslationNMTSentenceTranslationDual Graph Convolutional Network for Semantic Segmentation
Exploiting long-range contextual information is key for pixel-wise prediction tasks such as semantic segmentation. In contrast to previous work that uses multi-scale feature fusion or dilated convolutions, we propose a n…
Semantic SegmentationNever a Dull Moment: Distributional Properties as a Baseline for Time-Series Classification
The variety of complex algorithmic approaches for tackling time-series classification problems has grown considerably over the past decades, including the development of sophisticated but challenging-to-interpret deep-le…
ClassificationTime SeriesTime Series ClassificationContext as Prior: Bayesian-Inspired Intent Inference for Non-Speaking Agents with a Household Cat Testbed
Many agents in real-world environments cannot reliably communicate their goals through language, including household pets, pre-verbal infants, and other non-speaking embodied agents. In such settings, intent must be infe…