nGPT: Normalized Transformer with Representation Learning on the Hypersphere

2024-10-01 · Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, Boris Ginsburg

We propose a novel neural network architecture, the normalized Transformer (nGPT) with representation learning on the hypersphere. In nGPT, all vectors forming the embeddings, MLP, attention matrices and hidden states are unit norm normalized. The input stream of tokens travels on the surface of a hypersphere, with each layer contributing a displacement towards the target output predictions. These displacements are defined by the MLP and attention blocks, whose vector components also reside on the same hypersphere. Experiments show that nGPT learns much faster, reducing the number of training steps required to achieve the same accuracy by a factor of 4 to 20, depending on the sequence length.

📄 PDF Abstract BibTeX arXiv:2410.01131

Code (0)

등록된 구현이 없습니다.

Tasks

Representation Learning

Methods 이 논문이 사용한 방법론

Attention 설명 없음

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Multi-Head Attention 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Adam 설명 없음

Residual Connection 설명 없음

Position-Wise Feed-Forward Layer 설명 없음

Normalized Architectures are Natively 4-Bit

2026-05-07 · Maxim Fishman, Brian Chmiel, Ron Banner, Daniel Soudry 외 arxiv

Training large language models at 4-bit precision is critical for efficiency. We show that nGPT, an architecture that constrains weights and hidden representations to the unit hypersphere, is inherently more robust to lo…

Learning Rate Transfer in Normalized Transformers

2026-04-29 · Boris Shigida, Boris Hanin, Andrey Gromov arxiv

The Normalized Transformer, or nGPT (arXiv:2410.01131) achieves impressive training speedups and does not require weight decay or learning rate warmup. However, despite having hyperparameters that explicitly scale with m…

Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere

2020-05-20 · Tongzhou Wang, Phillip Isola

Contrastive representation learning has been outstandingly successful in practice. In this work, we identify two key properties related to the contrastive loss: (1) alignment (closeness) of features from positive pairs, …

Contrastive Learning Representation Learning

Cosine-Normalized Attention for Hyperspectral Image Classification

2026-04-02 · Muhammad Ahmad, Manuel Mazzara arxiv

Transformer-based methods have improved hyperspectral image classification (HSIC) by modeling long-range spatial-spectral dependencies; however, their attention mechanisms typically rely on dot-product similarity, which …

Hyperspectral Image Classification Representation Learning

Symmetry-Aware Transformer Training for Automated Planning

2025-08-11 · Markus Fritzsche, Elliot Gestrin, Jendrik Seipp arxiv

While transformers excel in many settings, their application in the field of automated planning is limited. Prior work like PlanGPT, a state-of-the-art decoder-only transformer, struggles with extrapolation from easy to …

Contrastive Learning