Implicit Bias of Mirror Flow on Separable Data
We examine the continuous-time counterpart of mirror descent, namely mirror flow, on classification problems which are linearly separable. Such problems are minimised at infinity' and have many possible solutions; we study which solution is preferred by the algorithm depending on the mirror potential. For exponential tailed losses and under mild assumptions on the potential, we show that the iterates converge in direction towards a $\phi_\infty$-maximum margin classifier. The function $\phi_\infty$ is the \textit{horizon function} of the mirror potential and characterises its shape at infinity'. When the potential is separable, a simple formula allows to compute this function. We analyse several examples of potentials and provide numerical experiments highlighting our results.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regression
We examine the implicit bias of mirror flow in univariate least squares error regression with wide and shallow neural networks. For a broad class of potential functions, we show that mirror flow exhibits lazy training an…
Characterizing Implicit Bias in Terms of Optimization Geometry
We study the implicit bias of generic optimization methods, such as mirror descent, natural gradient descent, and steepest descent with respect to different potentials and norms, when optimizing underdetermined linear re…
General ClassificationregressionImplicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent
As part of the effort to understand implicit bias of gradient descent in overparametrized models, several results have shown how the training trajectory on the overparametrized model can be understood as mirror descent o…
Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Implicit bias plays an important role in explaining how overparameterized models generalize well. Explicit regularization like weight decay is often employed in addition to prevent overfitting. While both concepts have b…
Faster Margin Maximization Rates for Generic and Adversarially Robust Optimization Methods
First-order optimization methods tend to inherently favor certain solutions over others when minimizing an underdetermined training objective that has multiple global optima. This phenomenon, known as implicit bias, play…
Binary Classification