NF4 Isn't Information Theoretically Optimal (and that's Good)
This note shares some simple calculations and experiments related to absmax-based blockwise quantization, as used in Dettmers et al., 2023. Their proposed NF4 data type is said to be information theoretically optimal for representing normally distributed weights. I show that this can't quite be the case, as the distribution of the values to be quantized depends on the block-size. I attempt to apply these insights to derive an improved code based on minimizing the expected L1 reconstruction error, rather than the quantile based method. This leads to improved performance for larger quantization block sizes, while both codes perform similarly at smaller block sizes.
Code (1)
Tasks
QuantizationSimilar Papers 제목 키워드 기반
Value Pursuit Iteration
Value Pursuit Iteration (VPI) is an approximate value iteration algorithm that finds a close to optimal policy for reinforcement learning and planning problems with large state spaces. VPI has two main features: First, …
Reinforcement LearningReinforcement Learning (RL)Near-Optimal Comparison Based Clustering
The goal of clustering is to group similar objects into meaningful partitions. This process is well understood when an explicit similarity measure between the objects is given. However, far less is known when this inform…
ClusteringMode Connectivity in Auction Design
Optimal auction design is a fundamental problem in algorithmic game theory. This problem is notoriously difficult already in very simple settings. Recent work in differentiable economics showed that neural networks can e…
Good Semi-supervised VAE Requires Tighter Evidence Lower Bound
Semi-supervised learning approaches based on generative models have now encountered 3 challenges: (1) The two-stage training strategy is not robust. (2) Good semi-supervised learning results and good generative performan…
4kA connection between bacterial chemotactic network and optimal filtering
The chemotactic network of Escherichia coli has been studied extensively both biophysically and information-theoretically. Nevertheless, the connection between these two aspects is still elusive. In this work, we report …