Optimization Landscapes of Wide Deep Neural Networks Are Benign
We analyze the optimization landscapes of deep learning with wide networks. We highlight the importance of constraints for such networks and show that constraint -- as well as unconstraint -- empirical-risk minimization over such networks has no confined points, that is, suboptimal parameters that are difficult to escape from. Hence, our theories substantiate the common belief that wide neural networks are not only highly expressive but also comparably easy to optimize.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Analysis of the Optimization Landscapes for Overcomplete Representation Learning
We study nonconvex optimization landscapes for learning overcomplete representations, including learning (i) sparsely used overcomplete dictionaries and (ii) convolutional dictionaries, where these unsupervised learning …
global-optimizationRepresentation LearningGeometric Analysis of Nonconvex Optimization Landscapes for Overcomplete Learning
Learning overcomplete representations finds many applications in machine learning and data analytics. In the past decade, despite the empirical success of heuristic methods, theoretical understandings and explanations of…
Representation LearningNesterov acceleration in benignly non-convex landscapes
While momentum-based optimization algorithms are commonly used in the notoriously non-convex optimization problems of deep learning, their analysis has historically been restricted to the convex and strongly convex setti…
Deep LearningNo Spurious Local Minima: on the Optimization Landscapes of Wide and Deep Neural Networks
Empirical studies suggest that wide neural networks are comparably easy to optimize, but mathematical support for this observation is scarce. In this paper, we analyze the optimization landscapes of deep learning with wi…
Deep LearningSearching in the Forest for Local Bayesian Optimization
Because of its sample efficiency, Bayesian optimization (BO) has become a popular approach dealing with expensive black-box optimization problems, such as hyperparameter optimization (HPO). Recent empirical experiments s…
Bayesian OptimizationHyperparameter Optimization