On The Mathematics of the Natural Physics of Optimization
A number of optimization algorithms have been inspired by the physics of Newtonian motion. Here, we ask the question: do algorithms themselves obey some `natural laws of motion,'' and can they be derived by an application of these laws? We explore this question by positing the theory that optimization algorithms may be considered as some manifestation of hidden algorithm primitives that obey certain universal non-Newtonian dynamics. This natural physics of optimization is developed by equating the terminal transversality conditions of an optimal control problem to the generalized Karush/John-Kuhn-Tucker conditions of an optimization problem. Through this equivalence formulation, the data functions of a given constrained optimization problem generate a natural vector field that permeates an entire hidden space with information on the optimality conditions. An action-at-a-distance'' operation via a Pontryagin-type minimum principle produces a local action to deliver a globalized result by way of a Hamilton-Jacobi inequality. An inverse-optimal algorithm is generated by performing control jumps that dissipate quantized `energy'' defined by a search Lyapunov function. Illustrative applications of the proposed theory show that a large number of algorithms can be generated and explained in terms of the new mathematical physics of optimization.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Linear Algebra and Duality of Neural Networks
Bases, mappings, projections and metrics, natural for Neural network training, are introduced. Graph-theoretical interpretation is offered. Non-Gaussianity naturally emerges, even in relatively simple datasets. Training …
Reality: Physics, the Physical Universe Language and Mathematics, Physics Formalisms in Human Brain Machine Language
The nature of the existence, revealed through Human cognitive system, has been evolving since the development of the languages. Part of such revelations were the geometrical forms and the numbers, whose beauty and order,…
Rigor with Machine Learning from Field Theory to the Poincaré Conjecture
Machine learning techniques are increasingly powerful, leading to many breakthroughs in the natural sciences, but they are often stochastic, error-prone, and blackbox. How, then, should they be utilized in fields such as…
Learning TheoryA Century of Controversy over the Foundations of Mathematics
This is the transcript of a lecture given at UMass-Lowell in which I compare and contrast the work of Godel and of Turing and my own work on incompleteness. I also discuss randomness in physics vs randomness in pure math…
The ORCA Benchmark: Evaluating Real-World Calculation Accuracy in Large Language Models
We present ORCA (Omni Research on Calculation in AI) Benchmark - a novel benchmark that evaluates large language models (LLMs) on multi-domain, real-life quantitative reasoning using verified outputs from Omni's calculat…
Domain Generalization