Generalized Momentum-Based Methods: A Hamiltonian Perspective
We take a Hamiltonian-based perspective to generalize Nesterov's accelerated gradient descent and Polyak's heavy ball method to a broad class of momentum methods in the setting of (possibly) constrained minimization in Euclidean and non-Euclidean normed vector spaces. Our perspective leads to a generic and unifying nonasymptotic analysis of convergence of these methods in both the function value (in the setting of convex optimization) and in norm of the gradient (in the setting of unconstrained, possibly nonconvex, optimization). Our approach relies upon a time-varying Hamiltonian that produces generalized momentum methods as its equations of motion. The convergence analysis for these methods is intuitive and is based on the conserved quantities of the time-dependent Hamiltonian.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Learning Generalized Hamiltonian Dynamics with Stability from Noisy Trajectory Data
We introduce a robust framework for learning various generalized Hamiltonian dynamics from noisy, sparse phase-space data and in an unsupervised manner based on variational Bayesian inference. Although conservative, diss…
Bayesian InferenceGaussian ProcessesGeneralized hybrid momentum maps and reduction by symmetries of forced mechanical systems with inelastic collisions
This paper discusses reduction by symmetries for autonomous and non-autonomous forced mechanical systems with inelastic collisions. In particular, we introduce the notion of generalized hybrid momentum map and hybrid con…
Data-Driven Identification of Quadratic Representations for Nonlinear Hamiltonian Systems using Weakly Symplectic Liftings
We present a framework for learning Hamiltonian systems using data. This work is based on a lifting hypothesis, which posits that nonlinear Hamiltonian systems can be written as nonlinear systems with cubic Hamiltonians.…
PositionSymplectic ODE-Net: Learning Hamiltonian Dynamics with Control
In this paper, we introduce Symplectic ODE-Net (SymODEN), a deep learning framework which can infer the dynamics of a physical system, given by an ordinary differential equation (ODE), from observed state trajectories. T…
Inductive BiasMove on Muon : A Hamiltonian probability gradient flow perspective of Muon optimizer
We develop a gradient flow on the space of probability measures defined on matrix-valued parameters induced by regularized Muon, an analytically smoothed version of the idealized Muon optimizer. The key observation is th…