Continuous Approximations for Improving Quantization Aware Training of LLMs

2024-10-06 · He Li, Jianhang Hong, Yuanzhuo Wu, Snehal Adbol, Zonglin Li

Model compression methods are used to reduce the computation and energy requirements for Large Language Models (LLMs). Quantization Aware Training (QAT), an effective model compression method, is proposed to reduce performance degradation after quantization. To further minimize this degradation, we introduce two continuous approximations to the QAT process on the rounding function, traditionally approximated by the Straight-Through Estimator (STE), and the clamping function. By applying both methods, the perplexity (PPL) on the WikiText-v2 dataset of the quantized model reaches 9.0815, outperforming 9.9621 by the baseline. Also, we achieve a 2.76% improvement on BoolQ, and a 5.47% improvement on MMLU, proving that the step sizes and weights can be learned more accurately with our approach. Our method achieves better performance with the same precision, model size, and training setup, contributing to the development of more energy-efficient LLMs technology that aligns with global sustainability goals.

📄 PDF Abstract BibTeX arXiv:2410.10849

Code (0)

등록된 구현이 없습니다.

Tasks

MMLU Model Compression Quantization

Methods 이 논문이 사용한 방법론

AWARE We propose to theoretically and empirically examine the effect of incorporating weighting schemes into walk-aggregating GNNs. To this end, we propose a simple, interpretable, and…

PTQTP: Post-Training Quantization to Trit-Planes for Large Language Models

2025-09-21 · He Xiao, Runming Yang, Qingyao Yang, Wendong Xu 외 arxiv

Post-training quantization (PTQ) of large language models (LLMs) to extremely low bit-widths remains challenging due to the fundamental trade-off between computational efficiency and representational capacity. While exis…

Computational Efficiency Mathematical Reasoning

L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models

2024-02-07 · Hyesung Jeon, Yulhwa Kim, Jae-Joon Kim

Due to the high memory and computational costs associated with large language models (LLMs), model compression techniques such as quantization, which reduces inference costs, and parameter-efficient fine-tuning (PEFT) me…

Few-Shot Learning In-Context Learning Model Compression parameter-efficient fine-tuning+1

EfficientQAT: Efficient Quantization-Aware Training for Large Language Models

2024-07-10 · Mengzhao Chen, Wenqi Shao, Peng Xu, Jiahao Wang 외

Large language models (LLMs) are crucial in modern natural language processing and artificial intelligence. However, they face challenges in managing their significant memory requirements. Although quantization-aware tra…

GPU Quantization

LLM-QAT: Data-Free Quantization Aware Training for Large Language Models

2023-05-29 · Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang 외

Several post-training quantization methods have been applied to large language models (LLMs), and have been shown to perform well down to 8-bits. We find that these methods break down at lower bit precision, and investig…

Data Free Quantization Quantization

Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models?

2025-02-17 · Jacob Nielsen, Peter Schneider-Kamp, Lukas Galke

Large language models (LLMs) require immense resources for training and inference. Quantization, a technique that reduces the precision of model parameters, offers a promising solution for improving LLM efficiency and su…

Quantization