Depth, Not Data: An Analysis of Hessian Spectral Bifurcation
The eigenvalue distribution of the Hessian matrix plays a crucial role in understanding the optimization landscape of deep neural networks. Prior work has attributed the well-documented ``bulk-and-spike'' spectral structure, where a few dominant eigenvalues are separated from a bulk of smaller ones, to the imbalance in the data covariance matrix. In this work, we challenge this view by demonstrating that such spectral Bifurcation can arise purely from the network architecture, independent of data imbalance. Specifically, we analyze a deep linear network setup and prove that, even when the data covariance is perfectly balanced, the Hessian still exhibits a Bifurcation eigenvalue structure: a dominant cluster and a bulk cluster. Crucially, we establish that the ratio between dominant and bulk eigenvalues scales linearly with the network depth. This reveals that the spectral gap is strongly affected by the network architecture rather than solely by data distribution. Our results suggest that both model architecture and data characteristics should be considered when designing optimization algorithms for deep networks.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Hessian Spectral Analysis at Foundation Model Scale
Accurate Hessian spectra of foundation models have remained out of reach, leading most prior work to rely on small models or strong structural approximations. We show that faithful spectral analysis of the true Hessian i…
Community detection with the Bethe-Hessian
The Bethe-Hessian matrix, introduced by Saade, Krzakala, and Zdeborov\'a (2014), is a Hermitian matrix designed for applying spectral clustering algorithms to sparse networks. Rather than employing a non-symmetric and hi…
Community DetectionStochastic Block ModelBifurcation of the neuronal population dynamics of the modified theta model: transition to macroscopic gamma oscillation
Interactions of inhibitory neurons produce gamma oscillations (30--80 Hz) in the local field potential, which is known to be involved in functions such as cognition and attention. In this study, the modified theta model …
On the Power-Law Hessian Spectrums in Deep Learning
It is well-known that the Hessian of deep loss landscape matters to optimization, generalization, and even robustness of deep learning. Recent works empirically discovered that the Hessian spectrum in deep learning has a…
Deep LearningOptimization and Generalization Guarantees for Weight Normalization
Weight normalization (WeightNorm) is widely used in practice for the training of deep neural networks and modern deep learning libraries have built-in implementations of it. In this paper, we provide the first theoretica…