From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models

2025-04-08 · Chejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, Bo Li, Bryan Catanzaro

Long-context capabilities are essential for a wide range of applications, including document and video understanding, in-context learning, and inference-time scaling, all of which require models to process and reason over long sequences of text and multimodal data. In this work, we introduce a efficient training recipe for building ultra-long context LLMs from aligned instruct model, pushing the boundaries of context lengths from 128K to 1M, 2M, and 4M tokens. Our approach leverages efficient continued pretraining strategies to extend the context window and employs effective instruction tuning to maintain the instruction-following and reasoning abilities. Our UltraLong-8B, built on Llama3.1-Instruct with our recipe, achieves state-of-the-art performance across a diverse set of long-context benchmarks. Importantly, models trained with our approach maintain competitive performance on standard benchmarks, demonstrating balanced improvements for both long and short context tasks. We further provide an in-depth analysis of key design choices, highlighting the impacts of scaling strategies and data composition. Our findings establish a robust framework for efficiently scaling context lengths while preserving general model capabilities. We release all model weights at: https://ultralong.github.io/.

📄 PDF Abstract BibTeX arXiv:2504.06214

Code (0)

등록된 구현이 없습니다.

Tasks

In-Context LearningInstruction FollowingVideo Understanding

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models

2025-11-28 · Xiang Hu, Zhanchao Zhou, Ruiqi Liang, Zehuan Li 외 arxiv

This work explores the challenge of building ``Machines that Can Remember'', framing long-term memory as the problem of efficient ultra-long context modeling. We argue that this requires three key properties: \textbf{spa…

MTraining: Distributed Dynamic Sparse Attention for Efficient Ultra-Long Context Training

2025-10-21 · Wenxuan Li, Chengruidong Zhang, Huiqiang Jiang, Yucheng Li 외 arxiv

The adoption of long context windows has become a standard feature in Large Language Models (LLMs), as extended contexts significantly enhance their capacity for complex reasoning and broaden their applicability across d…

UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models

2025-10-12 · Guangxin He, Shen Nie, Fengqi Zhu, Yuankang Zhao 외 arxiv

Diffusion LLMs have attracted growing interest, with plenty of recent work emphasizing their great potential in various downstream tasks; yet the long-context behavior of diffusion LLMs remains largely uncharted. We pres…

Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning

2026-06-12 · NVIDIA, :, Aaron Blakeman, Aaron Thomas 외 arxiv

We introduce Nemotron 3 Ultra, a 550 billion total and 55 billion active parameter Mixture-of-Experts Hybrid Mamba-Attention language model. We pre-trained Nemotron 3 Ultra on 20 trillion text tokens, then extended the c…

Reinforcement Learning

UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models' Reasoning Abilities

2025-07-26 · Dong Du, Shulin Liu, Tao Yang, Shaohua Chen 외 arxiv

Recent advances in large language models (LLMs) have highlighted the potential of reinforcement learning with verifiable rewards (RLVR) to enhance reasoning capabilities through extended output sequences. However, tradit…