Post Training Quantization of Large Language Models with Microscaling Formats

2024-05-12 · Sayeh Sharify, Utkarsh Saxena, Zifei Xu, Wanzin Yazar, Ilya Soloveychik, Xin Wang

Large Language Models (LLMs) have distinguished themselves with outstanding performance in complex language modeling tasks, yet they come with significant computational and storage challenges. This paper explores the potential of quantization to mitigate these challenges. We systematically study the combined application of three well-known post-training techniques, SmoothQuant, AWQ, and GPTQ, and provide a comprehensive analysis of their interactions and implications for advancing LLM quantization. We enhance the versatility of these methods by enabling quantization to microscaling (MX) formats, extending the applicability of these PTQ algorithms beyond their original fixed-point format targets. We show that combining different PTQ methods enables us to quantize models to 4-bit weights and 8-bit activations using the MXINT format with negligible accuracy loss compared to the uncompressed baseline.

📄 PDF Abstract BibTeX arXiv:2405.07135

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage Modelling Quantization

Methods 이 논문이 사용한 방법론

OPT OPT is a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters. The model uses an AdamW optimizer and weight decay of 0.1. It follows a linear…

LLaMA LLaMA is a collection of foundation language models ranging from 7B to 65B parameters. It is based on the transformer architecture with various improvements that were…

Benchmarking Post-Training Quantization of Large Language Models under Microscaling Floating Point Formats

2026-01-14 · Manyi Zhang, Ji-Fu Li, Zhongao Sun, Haoli Bai 외 arxiv

Microscaling Floating-Point (MXFP) has emerged as a promising low-precision format for large language models (LLMs). Despite various post-training quantization (PTQ) algorithms being proposed, they mostly focus on intege…

LATMiX: Learnable Affine Transformations for Microscaling Quantization of LLMs

2026-02-04 · Ofir Gordon, Lior Dikstein, Arnon Netzer, Idan Achituve 외 arxiv

Post-training quantization (PTQ) is a widely used approach for reducing the memory and compute costs of large language models (LLMs). Recent studies have shown that applying invertible transformations to activations can …

Search Your Block Floating Point Scales!

2026-05-12 · Tanmaey Gupta, Hayden Prairie, Xiaoxia Wu, Reyna Abhyankar 외 arxiv

Quantization has emerged as a standard technique for accelerating inference for generative models by enabling faster low-precision computations and reduced memory transfers. Recently, GPU accelerators have added first-cl…

Is Finer Better? The Limits of Microscaling Formats in Large Language Models

2026-01-26 · Andrea Fasoli, Monodeep Kar, Chi-Chun Liu, Swagath Venkataramani 외 arxiv

Microscaling data formats leverage per-block tensor quantization to enable aggressive model compression with limited loss in accuracy. Unlocking their potential for efficient training and inference necessitates hardware-…

Model Compression

Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization

2025-09-27 · Vage Egiazarian, Roberto L. Castro, Denis Kuznedelev, Andrei Panferov 외 arxiv

The recent hardware-accelerated microscaling 4-bit floating-point formats such as MXFP4 and NVFP4, supported on NVIDIA and AMD GPUs, promise to revolutionize large language model (LLM) inference. Yet, their practical ben…