Simple2Complex: Global Optimization by Gradient Descent
A method named simple2complex for modeling and training deep neural networks is proposed. Simple2complex train deep neural networks by smoothly adding more and more layers to the shallow networks, as the learning procedure going on, the network is just like growing. Compared with learning by end2end, simple2complex is with less possibility trapping into local minimal, namely, owning ability for global optimization. Cifar10 is used for verifying the superiority of simple2complex.
Code (0)
등록된 구현이 없습니다.
Tasks
global-optimizationSimilar Papers 제목 키워드 기반
No Spurious Local Minima in a Two Hidden Unit ReLU Network
Deep learning models can be efficiently optimized via stochastic gradient descent, but there is little theoretical evidence to support this. A key question in optimization is to understand when the optimization landscape…
Vocal Bursts Valence PredictionLearning to Learn without Gradient Descent by Gradient Descent
We learn recurrent neural network optimizers trained on simple synthetic functions by gradient descent. We show that these learned optimizers exhibit a remarkable degree of transfer in that they can be used to efficientl…
Bayesian Optimizationglobal-optimizationTraining Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioning
We study the training dynamics of gradient descent in a softmax self-attention layer trained to perform linear regression and show that a simple first-order optimization algorithm can converge to the globally optimal sel…
Combining resampling and reweighting for faithful stochastic optimization
Many machine learning and data science tasks require solving non-convex optimization problems. When the loss function is a sum of multiple terms, a popular method is the stochastic gradient descent. Viewed as a process f…
Computational chemistryStochastic OptimizationContinuous vs. Discrete Optimization of Deep Neural Networks
Existing analyses of optimization in deep learning are either continuous, focusing on (variants of) gradient flow, or discrete, directly treating (variants of) gradient descent. Gradient flow is amenable to theoretical a…
Computational EfficiencyDeep LearningLearning TheoryOpen-Ended Question Answering