DAQ: Density-Aware Post-Training Weight-Only Quantization For LLMs
Large language models (LLMs) excel in various tasks but face deployment challenges due to hardware constraints. We propose density-aware post-training weight-only quantization (DAQ), which has two stages: 1) density-centric alignment, which identifies the center of high-density weights and centers the dynamic range on this point to align high-density weight regions with floating-point high-precision regions; 2) learnable dynamic range adjustment, which adjusts the dynamic range by optimizing quantization parameters (i.e., scale and zero-point) based on the impact of weights on the model output. Experiments on LLaMA and LLaMA-2 show that DAQ consistently outperforms the best baseline method, reducing perplexity loss by an average of 22.8% on LLaMA and 19.6% on LLaMA-2. Our code is available at https://github.com/LuoYingSong/DAQ.
Code (1)
Tasks
QuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Amortized mean-shift interacting particles
Bayesian inference for inverse problems is run to evaluate integrals -- posterior expectations, tail probabilities, and risks -- across a stream of observations. The standard estimate averages the integrand over posterio…
Bayesian InferenceLightweight posterior construction for gravitational-wave catalogs with the Kolmogorov-Arnold network
Neural density estimation has seen widespread applications in the gravitational-wave (GW) data analysis, which enables real-time parameter estimation for compact binary coalescences and enhances rapid inference for subse…
Density EstimationDAQ: Delta-Aware Quantization for Post-Training LLM Weight Compression
We introduce Delta-Aware Quantization (DAQ), a data-free post-training quantization framework that preserves the knowledge acquired during post-training. Standard quantization objectives minimize reconstruction error but…
Dense or Sparse? Post-Packing Interconnection Analysis in FPGAs
Packing is a crucial step of FPGA design, directly impacting interconnect complexity, routing congestion, and overall performance. This paper presents a post-packing interconnect-aware analysis, illustrating how dense (s…
BayesQ: Uncertainty-Guided Bayesian Quantization
We present BayesQ, an uncertainty-guided post-training quantization framework that is the first to optimize quantization under the posterior expected loss. BayesQ fits a lightweight Gaussian posterior over weights (diago…