SKIM: Any-bit Quantization Pushing The Limits of Post-Training Quantization

2024-12-05 · Runsheng Bai, Bo Liu, Qiang Liu

Large Language Models (LLMs) exhibit impressive performance across various tasks, but deploying them for inference poses challenges. Their high resource demands often necessitate complex, costly multi-GPU pipelines, or the use of smaller, less capable models. While quantization offers a promising solution utilizing lower precision for model storage, existing methods frequently experience significant performance drops at lower precision levels. Additionally, they typically provide only a limited set of solutions at specific bit levels, many of which are extensively manually tuned. To address these challenges, we propose a new method called SKIM: Scaled K-means clustering wIth Mixed precision. Our approach introduces two novel techniques: 1. A greedy algorithm to solve approximately optimal bit allocation across weight channels, and 2. A trainable scaling vector for non-differentiable K-means clustering. These techniques substantially improve performance and can be adapted to any given bit. Notably, in terms of model perplexity, our method narrows the gap between 3-bit quantized LLaMA models and their full precision counterparts by 16.3% on average.

📄 PDF Abstract BibTeX arXiv:2412.04180

Code (0)

등록된 구현이 없습니다.

Tasks

ClusteringGPUQuantization

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

LLaMA LLaMA is a collection of foundation language models ranging from 7B to 65B parameters. It is based on the transformer architecture with various improvements that were…

k-Means Clustering k-Means Clustering is a clustering algorithm that divides a training set into $k$ different clusters of examples that are near each other. It works by initializing $k$…

BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction

2021-02-10 · ICLR 2021 1 · Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang 외

We study the challenging task of neural network quantization without end-to-end retraining, called Post-training Quantization (PTQ). PTQ usually requires a small subset of training data but produces less powerful quantiz…

image-classification Image Classification object-detection Object Detection+1

CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs

2024-05-27 · Haoyu Wang, Bei Liu, Hang Shao, Bo Xiao 외

Parameter quantization for Large Language Models (LLMs) has attracted increasing attentions recently in reducing memory costs and improving computational efficiency. Early approaches have been widely adopted. However, th…

Computational Efficiency Quantization

Pushing the Limits of Block Rotations in Post-Training Quantization

2026-01-29 · Sai Sanjeet, Ian Colbert, Pablo Monteagudo-Lago, Giuseppe Franco 외 arxiv

Recent post-training quantization (PTQ) methods have adopted block rotations to diffuse outliers prior to rounding. While this reduces the overhead of online full-vector rotations, the effect of block structure on outlie…

Achieving binary weight and activation for LLMs using Post-Training Quantization

2025-04-07 · Siqing Song, Chuang Wang, Ruiqi Wang, Yi Yang 외

Quantizing large language models (LLMs) to 1-bit precision significantly reduces computational costs, but existing quantization techniques suffer from noticeable performance degradation when using weight and activation p…

Quantization

TreeQ: Pushing the Quantization Boundary of Diffusion Transformer via Tree-Structured Mixed-Precision Search

2025-12-06 · Kaicheng Yang, Kaisen Yang, Baiting Wu, Xun Zhang 외 arxiv

Diffusion Transformers (DiTs) have emerged as a highly scalable and effective backbone for image generation, outperforming U-Net architectures in both scalability and performance. However, their real-world deployment rem…

Image Generation