Adding One Neuron Can Eliminate All Bad Local Minima
One of the main difficulties in analyzing neural networks is the non-convexity of the loss function which may have many bad local minima. In this paper, we study the landscape of neural networks for binary classification tasks. Under mild assumptions, we prove that after adding one special neuron with a skip connection to the output, or one special neuron per layer, every local minimum is a global minimum.
Code (0)
등록된 구현이 없습니다.
Tasks
AllBinary ClassificationGeneral ClassificationSimilar Papers 제목 키워드 기반
Elimination of All Bad Local Minima in Deep Learning
In this paper, we theoretically prove that adding one special neuron per output unit eliminates all suboptimal local minima of any deep neural network, for multi-class classification, binary classification, and regressio…
AllBinary ClassificationDeep LearningGeneral Classification+1Sub-Optimal Local Minima Exist for Neural Networks with Almost All Non-Linear Activations
Does over-parameterization eliminate sub-optimal local minima for neural networks? An affirmative answer was given by a classical result in [59] for 1-hidden-layer wide neural networks. A few recent works have extended t…
AllDeep Encryption: Protecting Pre-Trained Neural Networks with Confusion Neurons
Having consumed huge amounts of training data and computational resource, large-scale pre-trained models are often considered key assets of AI service providers. This raises an important problem: how to prevent these mod…
PositionAnnihilation of Spurious Minima in Two-Layer ReLU Networks
We study the optimization problem associated with fitting two-layer ReLU neural networks with respect to the squared loss, where labels are generated by a target network. Use is made of the rich symmetry structure to dev…
Vocal Bursts Valence PredictionEliminating all bad Local Minima from Loss Landscapes without even adding an Extra Unit
Recent work has noted that all bad local minima can be removed from neural network loss landscapes, by adding a single unit with a particular parameterization. We show that the core technique from these papers can be use…
All