From Density Matrices to Phase Transitions in Deep Learning: Spectral Early Warnings and Interpretability
A key problem in the modern study of AI is predicting and understanding emergent capabilities in models during training. Inspired by methods for studying reactions in quantum chemistry, we present the ``2-datapoint reduced density matrix". We show that this object provides a computationally efficient, unified observable of phase transitions during training. By tracking the eigenvalue statistics of the 2RDM over a sliding window, we derive two complementary signals: the spectral heat capacity, which we prove provides early warning of second-order phase transitions via critical slowing down, and the participation ratio, which reveals the dimensionality of the underlying reorganization. Remarkably, the top eigenvectors of the 2RDM are directly interpretable making it straightforward to study the nature of the transitions. We validate across four distinct settings: deep linear networks, induction head formation, grokking, and emergent misalignment. We then discuss directions for future work using the 2RDM.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Eigenvalue spectral properties of sparse random matrices obeying Dale's law
This paper examines the relationship between sparse random network architectures and neural network stability by examining the eigenvalue spectral distribution. Specifically, we generalise classical eigenspectral results…
Phase retrieval in high dimensions: Statistical and computational phase transitions
We consider the phase retrieval problem of reconstructing a $n$-dimensional real or complex signal $\mathbf{X}^{\star}$ from $m$ (possibly noisy) observations $Y_\mu = | \sum_{i=1}^n \Phi_{\mu i} X^{\star}_i/\sqrt{n}|$, …
RetrievalVocal Bursts Intensity PredictionSpectral phase transitions and trainability in neural network learning dynamics
The emergence of low-dimensional structures in the spectra of neural network weight matrices is a common empirical feature of trained models, but the dynamical origin of this phenomenon during learning remains an open pr…
Representation LearningPhase Transitions and a Model Order Selection Criterion for Spectral Graph Clustering
One of the longstanding open problems in spectral graph clustering (SGC) is the so-called model order selection problem: automated selection of the correct number of clusters. This is equivalent to the problem of finding…
ClusteringGraph ClusteringModel SelectionSpectral Graph ClusteringSpectral Kernel Dynamics via Maximum Caliber: Fixed Points, Geodesics, and Phase Transitions
We derive a closed-form geometric functional for kernel dynamics on finite graphs by applying the Maximum Caliber (MaxCal) variational principle to the spectral transfer function h(lambda) of the graph Laplacian eigenbas…