Improving Energy Conserving Descent for Machine Learning: Theory and Practice
We develop the theory of Energy Conserving Descent (ECD) and introduce ECDSep, a gradient-based optimization algorithm able to tackle convex and non-convex optimization problems. The method is based on the novel ECD framework of optimization as physical evolution of a suitable chaotic energy-conserving dynamical system, enabling analytic control of the distribution of results - dominated at low loss - even for generic high-dimensional problems with no symmetries. Compared to previous realizations of this idea, we exploit the theoretical control to improve both the dynamics and chaos-inducing elements, enhancing performance while simplifying the hyper-parameter tuning of the optimization algorithm targeted to different classes of problems. We empirically compare with popular optimization methods such as SGD, Adam and AdamW on a wide range of machine learning problems, finding competitive or improved performance compared to the best among them on each task. We identify limitations in our analysis pointing to possibilities for additional improvements.
Code (1)
Tasks
Learning TheoryMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Classical and Quantum Speedups for Non-Convex Optimization via Energy Conserving Descent
The Energy Conserving Descent (ECD) algorithm was recently proposed (De Luca & Silverstein, 2022) as a global non-convex optimization method. Unlike gradient descent, appropriately configured ECD dynamics escape strict l…
Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimization
We introduce a novel framework for optimization based on energy-conserving Hamiltonian dynamics in a strongly mixing (chaotic) regime and establish its key properties analytically and numerically. The prototype is a disc…
Benchmarking Energy-Conserving Neural Networks for Learning Dynamics from Data
The last few years have witnessed an increased interest in incorporating physics-informed inductive bias in deep learning frameworks. In particular, a growing volume of literature has been exploring ways to enforce energ…
BenchmarkingInductive BiasTime SeriesTime Series AnalysisStabilization of Energy-Conserving Gaits for Point-Foot Planar Bipeds
The problem of designing and stabilizing impact-free, energy-conserving gaits is considered for underactuated, point-foot planar bipeds. Virtual holonomic constraints are used to design energy-conserving gaits. A desired…
Pseudo-Hamiltonian Neural Networks with State-Dependent External Forces
Hybrid machine learning based on Hamiltonian formulations has recently been successfully demonstrated for simple mechanical systems, both energy conserving and not energy conserving. We introduce a pseudo-Hamiltonian for…
Hybrid Machine Learning