A Gaussian Comparison Theorem for Training Dynamics in Machine Learning
We study training algorithms with data following a Gaussian mixture model. For a specific family of such algorithms, we present a non-asymptotic result, connecting the evolution of the model to a surrogate dynamical system, which can be easier to analyze. The proof of our result is based on the celebrated Gordon comparison theorem. Using our theorem, we rigorously prove the validity of the dynamic mean-field (DMF) expressions in the asymptotic scenarios. Moreover, we suggest an iterative refinement scheme to obtain more accurate expressions in non-asymptotic scenarios. We specialize our theory to the analysis of training a perceptron model with a generic first-order (full-batch) algorithm and demonstrate that fluctuation parameters in a non-asymptotic domain emerge in addition to the DMF kernels.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
A Novel Gaussian Min-Max Theorem and its Applications
A celebrated result by Gordon allows one to compare the min-max behavior of two Gaussian processes if certain inequality conditions are met. The consequences of this result include the Gaussian min-max (GMT) and convex G…
Binary ClassificationGaussian ProcessesLearning-Based Robust Fixed-Time Terminal Sliding Mode Control
In this paper, we develop and analyze an integral fixed-time sliding mode control method for a scenario in which the system model is only partially known, utilizing Gaussian processes. We present two theorems on fixed-ti…
Gaussian ProcessesOptimal Transport for Machine Learners
Optimal Transport is a foundational mathematical theory that connects optimization, partial differential equations, and probability. It offers a powerful framework for comparing probability distributions and has recently…
TASI Lectures on Physics for Machine Learning
These notes are based on lectures I gave at TASI 2024 on Physics for Machine Learning. The focus is on neural network theory, organized according to network expressivity, statistics, and dynamics. I present classic resul…
Kolmogorov-Arnold NetworksGaussian kernels on non-simply-connected closed Riemannian manifolds are never positive definite
We show that the Gaussian kernel $\exp\left\{-\lambda d_g^2(\bullet, \bullet)\right\}$ on any non-simply-connected closed Riemannian manifold $(\mathcal{M},g)$, where $d_g$ is the geodesic distance, is not positive defin…