StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths

2026-01-27 · Tianyi Chen, Sihan Chen, Xiaoyi Qu, Dan Zhao, Ruomei Yan, Jongwoo Ko, Luming Liang, Pashmina Cameron arxiv

Quantization-aware training (QAT) is essential for deploying large models under strict memory and latency constraints, yet achieving stable and robust optimization at ultra-low bitwidths remains challenging. Common approaches based on the straight-through estimator (STE) or soft quantizers often suffer from gradient mismatch, instability, or high computational overhead. As such, we propose StableQAT, a unified and efficient QAT framework that stabilizes training in ultra low-bit settings via a novel, lightweight, and theoretically grounded surrogate for backpropagation derived from a discrete Fourier analysis of the rounding operator. StableQAT strictly generalizes STE as the latter arises as a special case of our more expressive surrogate family, yielding smooth, bounded, and inexpensive gradients that improve QAT training performance and stability across various hyperparameter choices. In experiments, StableQAT exhibits stable and efficient QAT at 2-4 bit regimes, demonstrating improved training stability, robustness, and superior performance with negligible training overhead against standard QAT techniques. Our code is available at https://github.com/microsoft/StableQAT.

📄 PDF Abstract BibTeX arXiv:2601.19320

Code (0)

등록된 구현이 없습니다.

Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformers

2023-02-23 · Minsoo Kim, Kyuhong Shim, Seongmin Park, Wonyong Sung 외

Pre-trained Transformer models such as BERT have shown great success in a wide range of applications, but at the cost of substantial increases in model complexity. Quantization-aware training (QAT) is a promising method …

Knowledge Distillation Quantization

DAQ: Channel-Wise Distribution-Aware Quantization for Deep Image Super-Resolution Networks

2020-12-21 · Cheeun Hong, Heewon Kim, Sungyong Baik, Junghun Oh 외

Quantizing deep convolutional neural networks for image super-resolution substantially reduces their computational costs. However, existing works either suffer from a severe performance drop in ultra-low precision of 4 o…

Image Super-Resolution Quantization Super-Resolution

Robust Ultra Low-Bit Post-Training Quantization via Stable Diagonal Curvature Estimate

2026-04-15 · Jaemin Kim, Sungkyun Kim, Junyeol Lee, Jiwon Seo arxiv

Large Language Models (LLMs) are widely used across many domains, but their scale makes deployment challenging. Post-Training Quantization (PTQ) reduces memory footprint without retraining by leveraging a small calibrati…

An Extra RMSNorm is All You Need for Fine Tuning to 1.58 Bits

2025-05-12 · Cody Steinmetz, Gavin Childress, Aaron Herbst, Gavin Jones 외

Large language models (LLMs) have transformed natural-language processing, yet their scale makes real-world deployment costly. Post-training quantization reduces memory and computation but often degrades accuracy, while …

All Knowledge Distillation Language Modeling Language Modelling+1

WINDQuant: Weight-Informed Neural Decision-Making for Global Mixed-Precision LLM Quantization

2026-05-26 · Phong Nam Huu Nguyen, Khoi M. Le, Cong-Duy T Nguyen, Anh Tuan Luu 외 arxiv

Quantization is an effective approach to reduce the memory footprint and inference cost of large language models (LLMs), yet maintaining performance in the ultra-low-bit regime remains challenging. Existing post-training…

Reinforcement Learning