The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
We apply state-of-the-art tools in modern high-dimensional numerical linear algebra to approximate efficiently the spectrum of the Hessian of modern deepnets, with tens of millions of parameters, trained on real data. Our results corroborate previous findings, based on small-scale networks, that the Hessian exhibits "spiked" behavior, with several outliers isolated from a continuous bulk. We decompose the Hessian into different components and study the dynamics with training and sample size of each term individually.
Code (2)
Similar Papers 제목 키워드 기반
Measuring the Spectrum of Deepnet Hessians
We apply state-of-the-art tools in modern high-dimensional numerical linear algebra to approximate efficiently the spectrum of the Hessian of modern deepnets, with tens of millions of parameters, trained on real data. We…
Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
We consider deep classifying neural networks. We expose a structure in the derivative of the logits with respect to the parameters of the model, which is used to explain the existence of outliers in the spectrum of the H…
Closing the Curvature Gap: Full Transformer Hessians and Their Implications for Scaling Laws
The lack of theoretical results for Layer Normalization and feedforward Hessians has left a gap in the study of Transformer optimization landscapes. We address this by deriving explicit second-order expressions for these…
Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
Numerous researchers recently applied empirical spectral analysis to the study of modern deep learning classifiers. We identify and discuss an important formal class/cross-class structure and show how it lies at the orig…
ArticlesDeep LearningHIP: Hessian Interatomic Potentials without derivatives
Molecular Hessians, the second derivatives of the potential energy, are fundamental to many workflows in computational chemistry. Usually, accurate Hessians are computationally expensive to calculate and scale poorly wit…