Porcupine Neural Networks: (Almost) All Local Optima are Global
Neural networks have been used prominently in several machine learning and statistics applications. In general, the underlying optimization of neural networks is non-convex which makes their performance analysis challenging. In this paper, we take a novel approach to this problem by asking whether one can constrain neural network weights to make its optimization landscape have good theoretical properties while at the same time, be a good approximation for the unconstrained one. For two-layer neural networks, we provide affirmative answers to these questions by introducing Porcupine Neural Networks (PNNs) whose weight vectors are constrained to lie over a finite set of lines. We show that most local optima of PNN optimizations are global while we have a characterization of regions where bad local optimizers may exist. Moreover, our theoretical and empirical results suggest that an unconstrained neural network can be approximated using a polynomially-large PNN.
Code (1)
Tasks
AllSimilar Papers 제목 키워드 기반
Porcupine Neural Networks: Approximating Neural Network Landscapes
Neural networks have been used prominently in several machine learning and statistics applications. In general, the underlying optimization of neural networks is non-convex which makes analyzing their performance challen…
The loss surface of deep and wide neural networks
While the optimization problem behind deep neural networks is highly non-convex, it is frequently observed in practice that training deep networks seems possible without getting stuck in suboptimal points. It has been ar…
Federated reinforcement learning for robot motion planning with zero-shot generalization
This paper considers the problem of learning a control policy for robot motion planning with zero-shot generalization, i.e., no data collection and policy adaptation is needed when the learned policy is deployed in new e…
Motion PlanningZero-shot GeneralizationGlobal Minimum Energy State Estimation for Embedded Nonlinear Systems with Symmetry
Choosing a nonlinear state estimator for an application often involves a trade-off between local optimality (such as provided by an extended Kalman filter) and (almost-/semi-) global asymptotic stability (such as provide…
State EstimationSmall nonlinearities in activation functions create bad local minima in neural networks
We investigate the loss surface of neural networks. We prove that even for one-hidden-layer networks with "slightest" nonlinearity, the empirical risks have spurious local minima in most cases. Our results thus indicate …