Online normalizer calculation for softmax

2018-05-08 · Maxim Milakov, Natalia Gimelshein

The Softmax function is ubiquitous in machine learning, multiple previous works suggested faster alternatives for it. In this paper we propose a way to compute classical Softmax with fewer memory accesses and hypothesize that this reduction in memory accesses should improve Softmax performance on actual hardware. The benchmarks confirm this hypothesis: Softmax accelerates by up to 1.3x and Softmax+TopK combined and fused by up to 5x.

📄 PDF Abstract BibTeX arXiv:1805.02867

Code (1)

NVIDIA/online-softmax 공식 구현

Tasks

BIG-bench Machine Learning

Methods 이 논문이 사용한 방법론

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

HiFA4: Training-Free 4-bit FlashAttention on Ascend HIF4 NPUs for LLM Inference

2026-07-05 · Hui Dong, Yanzhao Li, Jie Gao, Chunlu Li 외 arxiv

We present HiFA4, a post-training operator-level design that executes both QK^T and PV in FlashAttention as 4-bit HIF4 Cube GEMMs for LLM inference on Ascend NPUs, while maintaining the online softmax state in FP16. To o…

SSN: Learning Sparse Switchable Normalization via SparsestMax

2019-03-09 · CVPR 2019 6 · Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang 외

Normalization methods improve both optimization and generalization of ConvNets. To further boost performance, the recently-proposed switchable normalization (SN) provides a new perspective for deep learning: it learns to…

On the accuracy of self-normalized log-linear models

2015-06-12 · NeurIPS 2015 12 · Jacob Andreas, Maxim Rabinovich, Dan Klein, Michael. I. Jordan

Calculation of the log-normalizer is a major computational obstacle in applications of log-linear models with large output spaces. The problem of fast normalizer computation has therefore attracted significant attention …

Generalization Bounds

VFA: Relieving Vector Operations in Flash Attention with Global Maximum Pre-computation

2026-04-14 · Yupeng Sun, Yanzhao Li, Zhiqiang Zou, Bai Du 외 arxiv

FlashAttention-style online softmax enables exact attention computation with linear memory by streaming score tiles through on-chip memory and maintaining a running maximum and normalizer. However, as attention kernels a…

Legendre transformation and information geometry for the maximum entropy theory of ecology

2021-03-20 · Pedro Pessoa

Here I investigate some mathematical aspects of the maximum entropy theory of ecology (METE). In particular I address the geometrical structure of METE endowed by information geometry. As novel results, the macrostate en…