Nonlinear discretizations and Newton's method: characterizing stationary points of regression objectives
Second-order methods are emerging as promising alternatives to standard first-order optimizers such as gradient descent and ADAM for training neural networks. Though the advantages of including curvature information in computing optimization steps have been celebrated in the scientific machine learning literature, the only second-order methods that have been studied are quasi-Newton, meaning that the Hessian matrix of the objective function is approximated. Though one would expect only to gain from using the true Hessian in place of its approximation, we show that neural network training reliably fails when relying on exact curvature information. The failure modes provide insight both into the geometry of nonlinear discretizations as well as the distribution of stationary points in the loss landscape, leading us to question the conventional wisdom that the loss landscape is replete with local minima.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Continuation Newton methods with deflation techniques for global optimization problems
The global minimum point of an optimization problem is of interest in engineering fields and it is difficult to be found, especially for a nonconvex large-scale optimization problem. In this article, we consider a new me…
Evolutionary Algorithmsglobal-optimizationNewton Informed Neural Operator for Computing Multiple Solutions of Nonlinear Partials Differential Equations
Solving nonlinear partial differential equations (PDEs) with multiple solutions using neural networks has found widespread applications in various fields such as physics, biology, and engineering. However, classical neur…
Ill-Posedness and Optimization Geometry for Nonlinear Neural Network Training
In this work we analyze the role nonlinear activation functions play at stationary points of dense neural network training problems. We consider a generic least squares loss function training formulation. We show that th…
Provably Convergent Plug-and-Play Quasi-Newton Methods
Plug-and-Play (PnP) methods are a class of efficient iterative methods that aim to combine data fidelity terms and deep denoisers using classical optimization algorithms, such as ISTA or ADMM, with applications in invers…
DeblurringImage DeblurringSuper-ResolutionThe Landscape of Deep Learning Algorithms
This paper studies the landscape of empirical risk of deep neural networks by theoretically analyzing its convergence behavior to the population risk as well as its stationary points and properties. For an $l$-layer line…
Deep LearningGeneralization Bounds