Theoretical insights into the optimization landscape of over-parameterized shallow neural networks
In this paper we study the problem of learning a shallow artificial neural network that best fits a training data set. We study this problem in the over-parameterized regime where the number of observations are fewer than the number of parameters in the model. We show that with quadratic activations the optimization landscape of training such shallow neural networks has certain favorable characteristics that allow globally optimal models to be found efficiently using a variety of local search heuristics. This result holds for an arbitrary training data of input/output pairs. For differentiable activation functions we also show that gradient descent, when suitably initialized, converges at a linear rate to a globally optimal model. This result focuses on a realizable model where the inputs are chosen i.i.d. from a Gaussian distribution and the labels are generated according to planted weight coefficients.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Evolutionary Diversity Optimization and the Minimum Spanning Tree Problem
In the area of evolutionary computation the calculation of diverse sets of high-quality solutions to a given optimization problem has gained momentum in recent years under the term evolutionary diversity optimization. Th…
DiversityEvolutionary AlgorithmsGeometric Analysis of Nonconvex Optimization Landscapes for Overcomplete Learning
Learning overcomplete representations finds many applications in machine learning and data analytics. In the past decade, despite the empirical success of heuristic methods, theoretical understandings and explanations of…
Representation LearningOn the Optimization Landscape of Dynamic Output Feedback: A Case Study for Linear Quadratic Regulator
The convergence of policy gradient algorithms in reinforcement learning hinges on the optimization landscape of the underlying optimal control problem. Theoretical insights into these algorithms can often be acquired fro…
Decision MakingPolicy Gradient MethodsTheoretical characterisation of the Gauss-Newton conditioning in Neural Networks
The Gauss-Newton (GN) matrix plays an important role in machine learning, most evident in its use as a preconditioning matrix for a wide family of popular adaptive methods to speed up optimization. Besides, it can also p…
Matrix Sensing with Kernel Optimal Loss: Robustness and Optimization Landscape
In this paper we study how the choice of loss functions of non-convex optimization problems affects their robustness and optimization landscape, through the study of noisy matrix sensing. In traditional regression tasks,…