ROSAQ: Rotation-based Saliency-Aware Weight Quantization for Efficiently Compressing Large Language Models

2025-06-16 · Junho Yoon, Geom Lee, Donghyeon Jeon, Inho Kang, Seung-Hoon Na

Quantization has been widely studied as an effective technique for reducing the memory requirement of large language models (LLMs), potentially improving the latency time as well. Utilizing the characteristic of rotational invariance of transformer, we propose the rotation-based saliency-aware weight quantization (ROSAQ), which identifies salient channels in the projection feature space, not in the original feature space, where the projected "principal" dimensions are naturally considered as "salient" features. The proposed ROSAQ consists of 1) PCA-based projection, which first performs principal component analysis (PCA) on a calibration set and transforms via the PCA projection, 2) Salient channel dentification, which selects dimensions corresponding to the K-largest eigenvalues as salient channels, and 3) Saliency-aware quantization with mixed-precision, which uses FP16 for salient dimensions and INT3/4 for other dimensions. Experiment results show that ROSAQ shows improvements over the baseline saliency-aware quantization on the original feature space and other existing quantization methods. With kernel fusion, ROSAQ presents about 2.3x speed up over FP16 implementation in generating 256 tokens with a batch size of 64.

📄 PDF Abstract BibTeX arXiv:2506.13472

Code (0)

등록된 구현이 없습니다.

Tasks

Quantization

Methods 이 논문이 사용한 방법론

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

PCA Principle Components Analysis (PCA) is an unsupervised method primary used for dimensionality reduction within machine learning. PCA is calculated via a singular value…

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization

2026-03-09 · Yeonsik Park, Hyeonseong Kim, Seungkyu Choi arxiv

Post-training quantization (PTQ) has emerged as a prevailing technique for deploying large language models (LLMs) efficiently in terms of both memory and computation, across edge devices and server platforms. Existing PT…

Saliency-Aware Regularized Quantization Calibration for Large Language Models

2026-05-07 · Yanlong Zhao, Xiaoyuan Cheng, Huihang Liu, Baihua He 외 arxiv

Post-training quantization (PTQ) is an effective approach for deploying large language models (LLMs) under memory and latency constraints. Most existing PTQ methods determine quantization parameters by minimizing a layer…

Minimizing the Hidden Cost of Scales: Graph-Guided Ultra-Low-Bit Quantization for Large Language Models

2026-06-03 · Rayyan Abdalla, Amir Hussein, Min Wu, Dinesh Manocha arxiv

Post-training quantization (PTQ) is critical for the efficient deployment of large language models (LLMs). Recent ultra-low-bit PTQ methods rely on rigid weight-saliency assumptions or position heuristics, introducing su…

RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models

2025-02-13 · Quan Wei, Chung-Yiu Yau, Hoi-To Wai, Yang 외

Supervised fine-tuning is a standard method for adapting pre-trained large language models (LLMs) to downstream tasks. Quantization has been recently studied as a post-training technique for efficient LLM deployment. To …