Softmax Bias Correction for Quantized Generative Models

2023-09-04 · Nilesh Prasad Pandey, Marios Fournarakis, Chirag Patel, Markus Nagel

Post-training quantization (PTQ) is the go-to compression technique for large generative models, such as stable diffusion or large language models. PTQ methods commonly keep the softmax activation in higher precision as it has been shown to be very sensitive to quantization noise. However, this can lead to a significant runtime and power overhead during inference on resource-constraint edge devices. In this work, we investigate the source of the softmax sensitivity to quantization and show that the quantization operation leads to a large bias in the softmax output, causing accuracy degradation. To overcome this issue, we propose an offline bias correction technique that improves the quantizability of softmax without additional compute during deployment, as it can be readily absorbed into the quantization parameters. We demonstrate the effectiveness of our method on stable diffusion v1.5 and 125M-size OPT language model, achieving significant accuracy improvement for 8-bit quantized softmax.

📄 PDF Abstract BibTeX arXiv:2309.01729

Code (0)

등록된 구현이 없습니다.

Tasks

Language ModelingLanguage Modelling Quantization

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

OPT OPT is a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters. The model uses an AdamW optimizer and weight decay of 0.1. It follows a linear…

Diffusion Diffusion models generate samples by gradually removing noise from a signal, and their training objective can be expressed as a reweighted variational lower-bound…

Quantized Keys Steal Attention: Bias Correction for KV-Cache Compression in Video Diffusion

2026-05-25 · Tuna Tuncer, Felix Becker, Thomas Pfeil arxiv

Chunk-wise autoregressive video diffusion models rely on a KV cache of previously generated chunks to avoid redundant computation, but this cache quickly becomes a memory bottleneck as videos grow longer. Methods that qu…

Analytical Correction for Subsampling Bias in Drifting Models

2026-04-29 · Jiaru Zhang, Zeyun Deng, Juanwu Lu, Ziran Wang 외 arxiv

Drifting models are capable one-step generative models trained to follow a drifting field. The field combines attractive and repulsive softmax-weighted centroids over the data and current-generator distributions. In prac…

The Kernel Mixture Network: A Nonparametric Method for Conditional Density Estimation of Continuous Random Variables

2017-05-19 · Luca Ambrogioni, Umut Güçlü, Marcel A. J. van Gerven, Eric Maris

This paper introduces the kernel mixture network, a new method for nonparametric estimation of conditional probability densities using neural networks. We model arbitrarily complex conditional densities as linear combina…

Density Estimation

Semi-Relaxed Quantization with DropBits: Training Low-Bit Neural Networks via Bitwise Regularization

2021-01-01 · Jung Hyun Lee, Jihun Yun, Sung Ju Hwang, Eunho Yang

Network quantization, which aims to reduce the bit-lengths of the network weights and activations, has emerged as one of the key ingredients to reduce the size of neural networks for their deployments to resource-limited…

Quantization

Semi-Relaxed Quantization with DropBits: Training Low-Bit Neural Networks via Bit-wise Regularization

2019-11-29 · Jung Hyun Lee, Jihun Yun, Sung Ju Hwang, Eunho Yang