The Physical Systems Behind Optimization Algorithms
We use differential equations based approaches to provide some {\it \textbf{physics}} insights into analyzing the dynamics of popular optimization algorithms in machine learning. In particular, we study gradient descent, proximal gradient descent, coordinate gradient descent, proximal coordinate gradient, and Newton's methods as well as their Nesterov's accelerated variants in a unified framework motivated by a natural connection of optimization algorithms to physical systems. Our analysis is applicable to more general algorithms and optimization problems {\it \textbf{beyond}} convexity and strong convexity, e.g. Polyak-\L ojasiewicz and error bound conditions (possibly nonconvex).
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningSimilar Papers 제목 키워드 기반
Beyond backpropagation: bilevel optimization through implicit differentiation and equilibrium propagation
This paper reviews gradient-based techniques to solve bilevel optimization problems. Bilevel optimization is a general way to frame the learning of systems that are implicitly defined through a quantity that they minimiz…
Bilevel OptimizationStudy and Design of Reconfigurable Intelligent Surfaces
In this thesis, we introduce the fundamental equations behind the estimation of the link budget in a communications channel, highlighting the key limitations of conventional systems. Furthermore, we investigate the use o…
Stochastic Online Optimization for Cyber-Physical and Robotic Systems
We propose a novel gradient-based online optimization framework for solving stochastic programming problems that frequently arise in the context of cyber-physical and robotic systems. Our problem formulation accommodates…
A Realistic Dataset for the Smart Home Device Scheduling Problem for DCOPs
The field of Distributed Constraint Optimization has gained momentum in recent years thanks to its ability to address various applications related to multi-agent cooperation. While techniques to solve Distributed Constra…
SchedulingTowards a Systems Theory of Algorithms
Traditionally, numerical algorithms are seen as isolated pieces of code confined to an {\em in silico} existence. However, this perspective is not appropriate for many modern computational approaches in control, learning…
Decision Making