An Analytical Theory of Power Law Spectral Bias in the Learning Dynamics of Diffusion Models
We developed an analytical framework for understanding how the learned distribution evolves during diffusion model training. Leveraging the Gaussian equivalence principle, we derived exact solutions for the gradient-flow dynamics of weights in one- or two-layer linear denoiser settings with arbitrary data. Remarkably, these solutions allowed us to derive the generated distribution in closed form and its KL divergence through training. These analytical results expose a pronounced power-law spectral bias, i.e., for weights and distributions, the convergence time of a mode follows an inverse power law of its variance. Empirical experiments on both Gaussian and image datasets demonstrate that the power-law spectral bias remains robust even when using deeper or convolutional architectures. Our results underscore the importance of the data covariance in dictating the order and rate at which diffusion models learn different modes of the data, providing potential explanations for why earlier stopping could lead to incorrect details in image generative models.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Permutation-Invariant Spectral Learning via Dyson Diffusion
Diffusion models are central to generative modeling and have been adapted to graphs by diffusing adjacency matrix representations. The challenge of having up to $n!$ such representations for graphs with $n$ nodes is only…
Computational EfficiencyImplicit bias produces neural scaling laws in learning curves, from perceptrons to deep networks
Scaling laws in deep learning - empirical power-law relationships linking model performance to resource growth - have emerged as simple yet striking regularities across architectures, datasets, and tasks. These laws are …
Neural tangent kernel analysis of PINN for advection-diffusion equation
Physics-informed neural networks (PINNs) numerically approximate the solution of a partial differential equation (PDE) by incorporating the residual of the PDE along with its initial/boundary conditions into the loss fun…
Eigenvalues in microeconomics
Square matrices often arise in microeconomics, particularly in network models addressing applications from opinion dynamics to platform regulation. Spectral theory provides powerful tools for analyzing their properties. …
A Distributed Game-Theoretic Solution for Power Management in the Uplink of Cell-Free Systems
This paper investigates cell-free massive multiple input multiple output systems with a particular focus on uplink power allocation. In these systems, uplink power control is highly non-trivial, since a single user termi…
Management