Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients
Neural scaling laws describe how pre-training loss decays as power laws with training time, model size, and compute. This position paper argues that the exponents of these power laws are fixed by generic mechanisms: a one-third time scaling due to the strong nonlinearity of Softmax, an inverse width scaling due to representational superposition, and an inverse depth scaling due to ensemble averaging of Transformer layers. These mechanisms are robust to a wide range of data structures and architectural details, placing current large language models in a universality class with fixed exponents. The coefficients, however, are expected to be sensitive to data and architecture details, and directly determine practical quantities such as the optimal model shape and the compute-optimal frontier. We therefore argue that understanding the coefficients is the key to near-term performance improvements, and that a closer examination of the current universality class may reveal pathways to better universality classes.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Dropout Universality: Scaling Laws and Optimal Scheduling at the Edge-of-Chaos
We develop a mean-field theory of dropout as a perturbation of critical signal propagation at the edge of chaos, and show that it predicts a simple, no-cost change to standard practice: \emph{front-loaded} dropout schedu…
Subsampled directed-percolation models explain scaling relations experimentally observed in the brain
Recent experimental results on spike avalanches measured in the urethane-anesthetized rat cortex have revealed scaling relations that indicate a phase transition at a specific level of cortical firing rate variability. T…
Non-equilibrium relaxation in a stochastic lattice Lotka-Volterra model
We employ Monte Carlo simulations to study a stochastic Lotka-Volterra model on a two-dimensional square lattice with periodic boundary conditions. If the (local) prey carrying capacity is finite, there exists an extinct…
Topology-dependent coalescence controls scaling exponents in finite networks
Multiple studies of neural avalanches across different data modalities led to the prominent hypothesis that the brain operates near a critical point. The observed exponents often indicate the mean-field directed-percolat…
Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study
Prior classical-ML learning-curve work fits power laws to tree, linear, and kernel models on tabular data, but at small scale: typically one curve, one team, a handful of cells. We present a distributed classroom-scale r…