MXNorm: Reusing MXFP block scales for efficient tensor normalisation

2026-03-13 · Callum McLean, Luke Y. Prince, Alexandre Payot, Paul Balança, Carlo Luschi arxiv

Matrix multiplication performance has long been the major bottleneck to scaling deep learning workloads, which has stimulated the design of new accelerators that use increasingly low-precision number formats. However, improvements in matrix multiplication performance have far outstripped improvements in performance on reductions and elementwise computations, which are still being performed in higher precision. In this work, we propose MXNorm, a drop-in replacement for RMSNorm that estimates the RMS using only the block scales calculated as part of the MXFP8 cast and enables a 32x decrease in the size of reduction needed for normalization. We validate our approximation method on pre-training of Llama 3 models of 125M, 1B and 8B parameters, finding minimal loss of training accuracy compared to a baseline using RMSNorm with MXFP8 matmuls. We also show practical kernel speedups using only torch.compile of up to 2.4x for MXNorm over RMSNorm, corresponding to a 1.3% speedup in Llama 3 8B transformer layers in MXFP8 and a 2.6% speedup in NVFP4.

📄 PDF Abstract BibTeX arXiv:2603.13180

Code (0)

등록된 구현이 없습니다.

DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantization

2026-04-20 · Haokun Lin, Xinle Jia, Haobo Xu, Bingchen Yao 외 arxiv

The MXFP4 microscaling format, which partitions tensors into blocks of 32 elements sharing an E8M0 scaling factor, has emerged as a promising substrate for efficient LLM inference, backed by native hardware support on NV…

Unveiling the Potential of Quantization with MXFP4: Strategies for Quantization Error Reduction

2026-01-30 · Jatin Chhugani, Geonhwa Jeong, Bor-Yiing Su, Yunjie Pan 외 arxiv

Large Language Models (LLMs) have intensified the need for low-precision formats that enable efficient, large-scale inference. The Open Compute Project (OCP) Microscaling (MX) standard is attractive due to its favorable …

Diagnosing FP4 inference: a layer-wise and block-wise sensitivity analysis of NVFP4 and MXFP4

2026-03-05 · Musa Cim, Burak Topcu, Mahmut Taylan Kandemir arxiv

Quantization addresses the high resource demand for large language models (LLMs) by alleviating memory pressure and bandwidth congestion and providing significantly scaled compute power with a tolerable impact on accurac…

Recipes for Pre-training LLMs with MXFP8

2025-05-30 · Asit Mishra, Dusan Stosic, Simon Layton

Precision scaling - using fewer bits to represent model parameters and related tensors during pre-training - has emerged as a compelling technique for improving GPU efficiency without sacrificing accuracy. Microscaling (…

GPU

TORQ: Two-Level Orthogonal Rotation for MXFP4 Quantization

2026-05-19 · Zukang Xu, Xing Hu, Dawei Yang arxiv

As Large Language Models (LLMs) advance toward practical deployment, the Microscaling FP4 (MXFP4) format has emerged as a cornerstone for next-generation low-bit inference, owing to its ability to balance high dynamic ra…