Extended critical regimes of deep neural networks
Deep neural networks (DNNs) have been successfully applied to many real-world problems, but a complete understanding of their dynamical and computational principles is still lacking. Conventional theoretical frameworks for analysing DNNs often assume random networks with coupling weights obeying Gaussian statistics. However, non-Gaussian, heavy-tailed coupling is a ubiquitous phenomenon in DNNs. Here, by weaving together theories of heavy-tailed random matrices and non-equilibrium statistical physics, we develop a new type of mean field theory for DNNs which predicts that heavy-tailed weights enable the emergence of an extended critical regime without fine-tuning parameters. In this extended critical regime, DNNs exhibit rich and complex propagation dynamics across layers. We further elucidate that the extended criticality endows DNNs with profound computational advantages: balancing the contraction as well as expansion of internal neural representations and speeding up training processes, hence providing a theoretical guide for the design of efficient neural architectures.
Code (1)
Similar Papers 제목 키워드 기반
Emergence of Hopf bifurcation in an extended SIR dynamic
In this paper, the SIR dynamics is extended by considering another compartmental which represents hospitalization of the critical cases. So a system of differential equations with four blocks is considered when there is …
Online learning of Koopman operator using streaming data from different dynamical regimes
The paper presents a framework for online learning of the Koopman operator using streaming data. Many complex systems for which data-driven modeling and control are sought provide streaming sensor data, the abundance of …
A Stochastic Dynamical Theory of LLM Self-Adversariality: Modeling Severity Drift as a Critical Process
This paper introduces a continuous-time stochastic dynamical framework for understanding how large language models (LLMs) may self-amplify latent biases or toxicity through their own chain-of-thought reasoning. The model…
Temporal Complexity and Self-Organization in an Exponential Dense Associative Memory Model
Dense Associative Memory (DAM) models generalize the classical Hopfield model by incorporating n-body or exponential interactions that greatly enhance storage capacity. While the criticality of DAM models has been largel…
Multi-Phase Spacecraft Trajectory Optimization via Transformer-Based Reinforcement Learning
Autonomous spacecraft control for mission phases such as launch, ascent, stage separation, and orbit insertion remains a critical challenge due to the need for adaptive policies that generalize across dynamically distinc…
Reinforcement Learning