LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

2024-02-21 · Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, Mao Yang

Large context window is a desirable feature in large language models (LLMs). However, due to high fine-tuning costs, scarcity of long texts, and catastrophic values introduced by new token positions, current extended context windows are limited to around 128k tokens. This paper introduces LongRoPE that, for the first time, extends the context window of pre-trained LLMs to an impressive 2048k tokens, with up to only 1k fine-tuning steps at within 256k training lengths, while maintaining performance at the original short context window. This is achieved by three key innovations: (i) we identify and exploit two forms of non-uniformities in positional interpolation through an efficient search, providing a better initialization for fine-tuning and enabling an 8x extension in non-fine-tuning scenarios; (ii) we introduce a progressive extension strategy that first fine-tunes a 256k length LLM and then conducts a second positional interpolation on the fine-tuned extended LLM to achieve a 2048k context window; (iii) we readjust LongRoPE on 8k length to recover the short context window performance. Extensive experiments on LLaMA2 and Mistral across various tasks demonstrate the effectiveness of our method. Models extended via LongRoPE retain the original architecture with minor modifications to the positional embedding, and can reuse most pre-existing optimizations.

📄 PDF Abstract BibTeX arXiv:2402.13753

Code (1)

microsoft/longrope 공식 구현 pytorch

Tasks

LongRoPE2: Near-Lossless LLM Context Window Scaling

2025-02-27 · Ning Shang, Li Lyna Zhang, Siyuan Wang, Gaokai Zhang 외

LongRoPE2 is a novel approach that extends the effective context window of pre-trained large language models (LLMs) to the target length, while preserving the performance on the original shorter context window. This is a…

PSC: Extending Context Window of Large Language Models via Phase Shift Calibration

2025-05-18 · Wenqiao Zhu, Chao Xu, Lulu Wang, Jun Wu

Rotary Position Embedding (RoPE) is an efficient position encoding approach and is widely utilized in numerous large language models (LLMs). Recently, a lot of methods have been put forward to further expand the context …

16k Position

LongEmbed: Extending Embedding Models for Long Context Retrieval

2024-04-18 · Dawei Zhu, Liang Wang, Nan Yang, YiFan Song 외

Embedding models play a pivot role in modern NLP applications such as IR and RAG. While the context limit of LLMs has been pushed beyond 1 million tokens, embedding models are still confined to a narrow context window no…

4k 8k Benchmarking Position+2

CLEX: Continuous Length Extrapolation for Large Language Models

2023-10-25 · Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang 외

Transformer-based Large Language Models (LLMs) are pioneering advances in many natural language processing tasks, however, their exceptional capabilities are restricted within the preset context window of Transformer. Po…

4k Position

CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning

2026-01-21 · Zhiyuan Lu, Chenliang Li, Yingcheng Shi, Weizhou Shen 외 arxiv

While large language models now handle million-token contexts, their capacity for reasoning across entire document repositories remains largely untested. Existing benchmarks are inadequate, as they are mostly limited to …