Getting Free Bits Back from Rotational Symmetries in LLMs
Current methods for compressing neural network weights, such as decomposition, pruning, quantization, and channel simulation, often overlook the inherent symmetries within these networks and thus waste bits on encoding redundant information. In this paper, we propose a format based on bits-back coding for storing rotationally symmetric Transformer weights more efficiently than the usual array layout at the same floating-point precision. We evaluate our method on Large Language Models (LLMs) pruned by SliceGPT (Ashkboos et al., 2024) and achieve a 3-5% reduction in total bit usage for free across different model sizes and architectures without impacting model performance within a certain numerical precision.
Code (0)
등록된 구현이 없습니다.
Tasks
QuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Using Machine Learning to Detect Rotational Symmetries from Reflectional Symmetries in 2D Images
Automated symmetry detection is still a difficult task in 2021. However, it has applications in computer vision, and it also plays an important part in understanding art. This paper focuses on aiding the latter by compar…
BIG-bench Machine LearningSymmetry DetectionEquivariant Representation Learning in the Presence of Stabilizers
We introduce Equivariant Isomorphic Networks (EquIN) -- a method for learning representations that are equivariant with respect to general group actions over data. Differently from existing equivariant representation lea…
Representation LearningSymmetry Detection and Classification in Drawings of Graphs
Symmetry is a key feature observed in nature (from flowers and leaves, to butterflies and birds) and in human-made objects (from paintings and sculptures, to manufactured objects and architectural design). Rotational, tr…
BIG-bench Machine LearningClassificationGeneral ClassificationSymmetry DetectionRotationally Equivariant Super-Resolution of Velocity Fields in Two-Dimensional Fluids Using Convolutional Neural Networks
This paper investigates the super-resolution (SR) of velocity fields in two-dimensional fluids from the viewpoint of rotational equivariance. SR refers to techniques that estimate high-resolution images from those in low…
Super-ResolutionTranslationLagrangian Neural Network with Differentiable Symmetries and Relational Inductive Bias
Realistic models of physical world rely on differentiable symmetries that, in turn, correspond to conservation laws. Recent works on Lagrangian and Hamiltonian neural networks show that the underlying symmetries of a sys…
Inductive Bias