QUAD: Quantization and Parameter-Efficient Tuning of LLM with Activation Decomposition

2025-03-25 · Yuxuan Hu, Xiaodong Chen, Cuiping Li, Hong Chen, Jing Zhang

Large Language Models (LLMs) excel in diverse applications but suffer inefficiency due to massive scale. While quantization reduces computational costs, existing methods degrade accuracy in medium-sized LLMs (e.g., Llama-3-8B) due to activation outliers. To address this, we propose QUAD (Quantization with Activation Decomposition), a framework leveraging Singular Value Decomposition (SVD) to suppress activation outliers for effective 4-bit quantization. QUAD estimates activation singular vectors offline using calibration data to construct an orthogonal transformation matrix P, shifting outliers to additional dimensions in full precision while quantizing rest components to 4-bit. Additionally, QUAD enables parameter-efficient fine-tuning via adaptable full-precision outlier weights, narrowing the accuracy gap between quantized and full-precision models. Experiments demonstrate that QUAD achieves 94% ~ 96% accuracy under W4A4 quantization and 98% accuracy with W4A4/A8 and parameter-efficient fine-tuning for Llama-3 and Qwen-2.5 models. Our code is available at \href{https://github.com/hyx1999/Quad}{repository}.

📄 PDF Abstract BibTeX arXiv:2503.19353

Code (1)

hyx1999/quad 공식 구현 pytorch

Tasks

parameter-efficient fine-tuning Quantization

Quadapter: Adapter for GPT-2 Quantization

2022-11-30 · Minseop Park, Jaeseong You, Markus Nagel, Simyung Chang

Transformer language models such as GPT-2 are difficult to quantize because of outliers in activations leading to a large quantization error. To adapt to the error, one must use quantization-aware training, which entails…

Quantization

Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

2019-09-12 · Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma 외

Transformer based architectures have become de-facto models used for a range of Natural Language Processing tasks. In particular, the BERT based models achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuA…

Linguistic Acceptability Natural Language Inference Quantization Semantic Textual Similarity+2

RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization

2024-07-10 · Xijie Huang, Zechun Liu, Shih-Yang Liu, Kwang-Ting Cheng

Low-Rank Adaptation (LoRA), as a representative Parameter-Efficient Fine-Tuning (PEFT)method, significantly enhances the training efficiency by updating only a small portion of the weights in Large Language Models (LLMs)…

parameter-efficient fine-tuning Quantization

QuantLRM: Quantization of Large Reasoning Models via Fine-Tuning Signals

2026-01-31 · Nan Zhang, Eugene Kwek, Yusen Zhang, Muyu Pan 외 arxiv

Weight-only quantization is important for compressing Large Language Models (LLMs). Inspired by the spirit of classical magnitude pruning, we study whether the magnitude of weight updates during reasoning-incentivized fi…

Reinforcement Learning Temporal Sequences

Enhancing Post-Training Quantization via Future Activation Awareness

2026-01-28 · Zheqi Lv, Zhenxuan Fan, Qi Tian, Wenqiao Zhang 외 arxiv

Post-training quantization (PTQ) is a widely used method to compress large language models (LLMs) without fine-tuning. It typically sets quantization hyperparameters (e.g., scaling factors) based on current-layer activat…