Fokker-Planck to Callan-Symanzik: evolution of weight matrices under training
The dynamical evolution of a neural network during training has been an incredibly fascinating subject of study. First principal derivation of generic evolution of variables in statistical physics systems has proved useful when used to describe training dynamics conceptually, which in practice means numerically solving equations such as Fokker-Planck equation. Simulating entire networks inevitably runs into the curse of dimensionality. In this paper, we utilize Fokker-Planck to simulate the probability density evolution of individual weight matrices in the bottleneck layers of a simple 2-bottleneck-layered auto-encoder and compare the theoretical evolutions against the empirical ones by examining the output data distributions. We also derive physically relevant partial differential equations such as Callan-Symanzik and Kardar-Parisi-Zhang equations from the dynamical equation we have.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Quantum Dynamics of Optimization Problems
In this letter, by establishing the Schr\"odinger equation of the optimization problem, the optimization problem is transformed into a constrained state quantum problem with the objective function as the potential energy…
An Analysis of Regularization and Fokker-Planck Residuals in Diffusion Models for Image Generation
Recent work has shown that diffusion models trained with the denoising score matching (DSM) objective often violate the Fokker--Planck (FP) equation that governs the evolution of the true data density. Directly penalizin…
Image GenerationA `Rosetta stone' for the population dynamics of spiking neuron networks
Populations of spiking neuron models have densities of their microscopic variables (e.g., single-cell membrane potentials) whose evolution fully capture the collective dynamics of biological networks, even outside equili…
Measuring inequality in society-oriented Lotka--Volterra-type kinetic equations
We present a possible approach to measuring inequality in a system of coupled Fokker-Planck-type equations that describe the evolution of distribution densities for two populations interacting pairwise due to social and/…
Local Fokker--Planck Geometry for Score Estimation: Heat-Ball Mean-Value Representations and Exact High-Dimensional Sampling
Score-based generative models and Langevin samplers rely on estimating the score function $\nabla_x\log p_t(x)$ of a forward diffusion. Classically this is tractable when the drift is linear: the marginal density is Gaus…