Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectives
We analyze the convergence rate of various momentum-based optimization algorithms from a dynamical systems point of view. Our analysis exploits fundamental topological properties, such as the continuous dependence of iterates on their initial conditions, to provide a simple characterization of convergence rates. In many cases, closed-form expressions are obtained that relate algorithm parameters to the convergence rate. The analysis encompasses discrete time and continuous time, as well as time-invariant and time-variant formulations, and is not limited to a convex or Euclidean setting. In addition, the article rigorously establishes why symplectic discretization schemes are important for momentum-based optimization algorithms, and provides a characterization of algorithms that exhibit accelerated convergence.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Conformal Symplectic and Relativistic Optimization
Arguably, the two most popular accelerated or momentum-based optimization methods in machine learning are Nesterov's accelerated gradient and Polyaks's heavy ball, both corresponding to different discretizations of a par…
FrictionSymplectic Momentum Neural Networks -- Using Discrete Variational Mechanics as a prior in Deep Learning
With deep learning gaining attention from the research community for prediction and control of real physical systems, learning important representations is becoming now more than ever mandatory. It is of extreme importan…
CaLiSym: Learning Symplectic Dynamics of Real-World Systems through Structured Canonical Lifts
Physics-informed learning promises data-efficient and stable dynamics prediction, yet its strongest geometric guarantees have largely remained confined to closed conservative systems. This excludes many robotic systems o…
Momentum Attention: The Physics of In-Context Learning and Spectral Forensics for Mechanistic Interpretability
The Mechanistic Interpretability (MI) program has mapped the Transformer as a precise computational graph. We extend this graph with a conservation law and time-varying AC dynamics, viewing it as a physical circuit. We i…
Symplectic Geometric Methods for Matrix Differential Equations Arising from Inertial Navigation Problems
This article explores some geometric and algebraic properties of the dynamical system which is represented by matrix differential equations arising from inertial navigation problems, such as the symplecticity and the ort…