Efficient Pretraining Length Scaling

2025-04-21 · Bohong Wu, Shen Yan, Sijun Zhang, Jianqiao Lu, Yutao Zeng, Ya Wang, Xun Zhou

Recent advances in large language models have demonstrated the effectiveness of length scaling during post-training, yet its potential in pre-training remains underexplored. We present the Parallel Hidden Decoding Transformer (\textit{PHD}-Transformer), a novel framework that enables efficient length scaling during pre-training while maintaining inference efficiency. \textit{PHD}-Transformer achieves this through an innovative KV cache management strategy that distinguishes between original tokens and hidden decoding tokens. By retaining only the KV cache of original tokens for long-range dependencies while immediately discarding hidden decoding tokens after use, our approach maintains the same KV cache size as the vanilla transformer while enabling effective length scaling. To further enhance performance, we introduce two optimized variants: \textit{PHD-SWA} employs sliding window attention to preserve local dependencies, while \textit{PHD-CSWA} implements chunk-wise sliding window attention to eliminate linear growth in pre-filling time. Extensive experiments demonstrate consistent improvements across multiple benchmarks.

📄 PDF Abstract BibTeX arXiv:2504.14992

Code (0)

등록된 구현이 없습니다.

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Multi-Head Attention 설명 없음

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Adam 설명 없음

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Position-Wise Feed-Forward Layer 설명 없음

Data Engineering for Scaling Language Models to 128K Context

2024-02-15 · Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue 외

We study the continual pretraining recipe for scaling language models' context lengths to 128K, with a focus on data engineering. We hypothesize that long context modeling, in particular \textit{the ability to utilize in…

4k Continual Pretraining

Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models

2026-07-09 · Aiwei Liu, Cheng Shi, Chuhan Wu, Ci Lei 외 arxiv

Scaling Large Language Models (LLMs) has been driven mainly by enlarging the Transformer backbone, but for an already-strong model this requires another round of costly pretraining. We study whether an existing backbone …

Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings

2025-12-13 · Yoav Gelberg, Koshi Eguchi, Takuya Akiba, Edoardo Cetin arxiv

So far, expensive finetuning beyond the pretraining sequence length has been a requirement for effectively extending the context of language models (LM). In this work, we break this key bottleneck by Dropping the Positio…

Asymptotic theory of in-context learning by linear attention

2024-05-20 · Yue M. Lu, Mary I. Letey, Jacob A. Zavatone-Veth, Anindita Maiti 외

Transformers have a remarkable ability to learn and execute tasks based on examples provided within the input itself, without explicit prior training. It has been argued that this capability, known as in-context learning…

Diversity In-Context Learning Memorization

Pretraining with Token-Level Adaptive Latent Chain-of-Thought

2026-02-09 · Boyi Zeng, Yiqin Hao, He Li, Shixiang Song 외 arxiv

Scaling large language models by increasing parameters and training data is increasingly constrained by limited high-quality corpora and rising communication costs. This work explores an alternative axis: increasing per-…