On the Implicit Bias in Deep-Learning Algorithms
Gradient-based deep-learning algorithms exhibit remarkable performance in practice, but it is not well-understood why they are able to generalize despite having more parameters than training examples. It is believed that implicit bias is a key factor in their ability to generalize, and hence it was widely studied in recent years. In this short survey, we explain the notion of implicit bias, review main results and discuss their implications.
Code (1)
Tasks
Deep LearningSurveySimilar Papers 제목 키워드 기반
Faster Margin Maximization Rates for Generic and Adversarially Robust Optimization Methods
First-order optimization methods tend to inherently favor certain solutions over others when minimizing an underdetermined training objective that has multiple global optima. This phenomenon, known as implicit bias, play…
Binary ClassificationFlavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networks
We study the implicit bias of the general family of steepest descent algorithms with infinitesimal learning rate in deep homogeneous neural networks. We show that: (a) an algorithm-dependent geometric margin starts incre…
On Implicit Bias in Overparameterized Bilevel Optimization
Many problems in machine learning involve bilevel optimization (BLO), including hyperparameter optimization, meta-learning, and dataset distillation. Bilevel problems consist of two nested sub-problems, called the outer …
Bilevel OptimizationDataset DistillationHyperparameter OptimizationMeta-LearningThe Rich and the Simple: On the Implicit Bias of Adam and SGD
Adam is the de facto optimization algorithm for several deep learning applications, but an understanding of its implicit bias and how it differs from other algorithms, particularly standard first-order methods such as (s…
Binary ClassificationOn the implicit minimization of alternative loss functions when training deep networks
Understanding the implicit bias of optimization algorithms is important in order to improve generalization of neural networks. One approach to try to exploit such understanding would be to then make the bias explicit in …
Inductive Bias