The Emergence of Spectral Universality in Deep Networks
Recent work has shown that tight concentration of the entire spectrum of singular values of a deep network's input-output Jacobian around one at initialization can speed up learning by orders of magnitude. Therefore, to guide important design choices, it is important to build a full theoretical understanding of the spectra of Jacobians at initialization. To this end, we leverage powerful tools from free probability theory to provide a detailed analytic understanding of how a deep network's Jacobian spectrum depends on various hyperparameters including the nonlinearity, the weight and bias distributions, and the depth. For a variety of nonlinearities, our work reveals the emergence of new universal limiting spectral distributions that remain concentrated around one even as the depth goes to infinity.
Code (1)
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Fear Universality and Doubt in Asset price movements
We take a look the changes of different asset prices over variable periods, using both traditional and spectral methods, and discover universality phenomena which hold (in some cases) across asset classes.
Persistence diagrams of random matrices via Morse theory: universality and a new spectral diagnostic
We prove that the persistence diagram of the sublevel set filtration of the quadratic form f(x) = x^T M x restricted to the unit sphere S^{n-1} is analytically determined by the eigenvalues of the symmetric matrix M. By …
How Universal is Metonymy? Results from a Large-Scale Multilingual Analysis
Metonymy is regarded by most linguists as a universal cognitive phenomenon, especially since the emergence of the theory of conceptual mappings. However, the field data backing up claims of universality has not been larg…
Emergence via Phase Transitions: Mechanism Landscapes and Universal Convergence Across Complex Systems
Across machine learning, biology, and physics, independently evolving systems often converge toward strikingly similar high-level structures despite radically different microscopic details. Grokking circuits converge acr…
Dynamics of genetic code evolution: The emergence of universality
We study the dynamics of genetic code evolution. The model of Vetsigian et al. [1] and Vetsigian [2] uses the mechanism of horizontal gene transfer to demonstrate convergence of the genetic code to a near universal solut…