On neural network kernels and the storage capacity problem
In this short note, we reify the connection between work on the storage capacity problem in wide two-layer treelike neural networks and the rapidly-growing body of literature on kernel limits of wide neural networks. Concretely, we observe that the "effective order parameter" studied in the statistical mechanics literature is exactly equivalent to the infinite-width Neural Network Gaussian Process Kernel. This correspondence connects the expressivity and trainability of wide two-layer neural networks.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Kernel Logistic Regression Learning for High-Capacity Hopfield Networks
Hebbian learning limits Hopfield network storage capacity (pattern-to-neuron ratio around 0.14). We propose Kernel Logistic Regression (KLR) learning. Unlike linear methods, KLR uses kernels to implicitly map patterns to…
regressionVirtual Energy Storage Sharing and Capacity Allocation
Energy storage can play an important role in energy management of end users. To promote an efficient utilization of energy storage, we develop a novel business model to enable virtual storage sharing among a group of use…
energy managementManagementThe Transmission Value of Energy Storage and Fundamental Limitations
This study addresses the transmission value of energy storage in electric grids. The inherent connection between storage and transmission infrastructure is captured from a "cumulative energy" perspective, which enables t…
BiSCo-LLM: Lookup-Free Binary Spherical Coding for Extreme Low-Bit Large Language Model Compression
Large language models (LLMs) are increasingly constrained by memory capacity, weight bandwidth, and checkpoint storage during deployment. Existing low-bit compression methods mainly follow two directions. Scalar or group…
Model CompressionDistributed storage algorithms with optimal tradeoffs
One of the primary objectives of a distributed storage system is to reliably store large amounts of source data for long durations using a large number $N$ of unreliable storage nodes, each with $c$ bits of storage capac…