QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression

2024-08-01 · Wenshan Wang, Yihang Wang, Yixing Fan, Huaming Liao, Jiafeng Guo

In-context learning (ICL) capabilities are foundational to the success of large language models (LLMs). Recently, context compression has attracted growing interest since it can largely reduce reasoning complexities and computation costs of LLMs. In this paper, we introduce a novel Query-gUIded aTtention cOmpression (QUITO) method, which leverages attention of the question over the contexts to filter useless information. Specifically, we take a trigger token to calculate the attention distribution of the context in response to the question. Based on the distribution, we propose three different filtering methods to satisfy the budget constraints of the context length. We evaluate the QUITO using two widely-used datasets, namely, NaturalQuestions and ASQA. Experimental results demonstrate that QUITO significantly outperforms established baselines across various datasets and downstream LLMs, underscoring its effectiveness. Our code is available at https://github.com/Wenshansilvia/attention_compressor.

📄 PDF Abstract BibTeX arXiv:2408.00274

Code (1)

wenshansilvia/attention_compressor 공식 구현 pytorch

Tasks

In-Context Learning

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Attention 설명 없음

Natural Language based Context Modeling and Reasoning for Ubiquitous Computing with Large Language Models: A Tutorial

2023-09-24 · Haoyi Xiong, Jiang Bian, Sijia Yang, Xiaofei Zhang 외

Large language models (LLMs) have become phenomenally surging, since 2018--two decades after introducing context-awareness into computing systems. Through taking into account the situations of ubiquitous devices, users a…

Natural Language Understanding Scheduling

Understanding the Logical Capabilities of Large Language Models via Out-of-Context Representation Learning

2025-03-13 · Jonathan Shaki, Emanuele La Malfa, Michael Wooldridge, Sarit Kraus

We study the capabilities of Large Language Models (LLM) on binary relations, a ubiquitous concept in math employed in most reasoning, math and logic benchmarks. This work focuses on equality, inequality, and inclusion, …

In-Context Learning Math Representation Learning

TeleMem: Building Long-Term and Multimodal Memory for Agentic AI

2025-12-12 · Chunliang Chen, Ming Guan, Xiao Lin, Jiaxu Li 외 arxiv

Large language models (LLMs) excel at many NLP tasks but struggle to sustain long-term interactions due to limited attention over extended dialogue histories. Retrieval-augmented generation (RAG) mitigates this issue but…

Multimodal Reasoning

SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verification

2025-12-02 · Zhendong Tan, Xingjun Zhang, Chaoyi Hu, Junjie Peng 외 arxiv

Growing demands from tasks like code generation, deep reasoning, and long-document understanding have made long-context generation a crucial capability for large language models (LLMs). Speculative decoding is one of the…

Code Generation

LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding

2026-02-04 · Gang Lin, Dongfang Li, Zhuoen Chen, Yukun Shi 외 arxiv

The proliferation of long-context large language models (LLMs) exposes a key bottleneck: the rapidly expanding key-value cache during decoding, which imposes heavy memory and latency costs. While recent approaches attemp…

Long-Context Understanding