Quantization robustness from dense representations of sparse functions in high-capacity kernel associative memory
High-capacity associative memories based on Kernel Logistic Regression (KLR) achieve strong retrieval performance but typically require substantial computational resources. This paper investigates the compressibility of KLR Hopfield networks to clarify the geometric principles underlying their robust representations. We present a geometric interpretation based on spontaneous symmetry breaking and Walsh analysis, and examine it through compression experiments involving quantization and pruning. The experiments reveal a clear asymmetry: the network remains robust under low-precision quantization while exhibiting strong sensitivity to pruning. We interpret this behavior through a "sparse function, dense representation" principle, in which a sparse input mapping is implemented through a dense bimodal parameterization. These findings suggest a practical route toward hardware-efficient kernel associative memories and provide insight into the geometric principles underlying robust representation in neural systems.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations
Generative models have recently gained attention in recommendation systems by directly predicting item identifiers from user interaction sequences. However, existing methods suffer from significant information loss due t…
QuantizationRecommendation SystemsRetrievalHow Can We Be So Dense? The Benefits of Using Highly Sparse Representations
Most artificial networks today rely on dense representations, whereas biological networks rely on sparse representations. In this paper we show how sparse representations can be more robust to noise and interference, as …
Computational EfficiencySparseKAN: Compressing Kolmogorov--Arnold Networks Across Basis Functions, Neurons, and Bits
Kolmogorov--Arnold Networks (KANs) replace scalar edge weights with learnable univariate functions parameterized by multiple basis coefficients. This introduces a source of redundancy that conventional neural-network com…
Minimizing FLOPs to Learn Efficient Sparse Representations
Deep representation learning has become one of the most widely adopted approaches for visual search, recommendation, and identification. Retrieval of such representations from a large database is however computationally …
QuantizationRepresentation LearningRetrievalRobust Depth Completion with Uncertainty-Driven Loss Functions
Recovering a dense depth image from sparse LiDAR scans is a challenging task. Despite the popularity of color-guided methods for sparse-to-dense depth completion, they treated pixels equally during optimization, ignoring…
Depth Completion