Implicitly Maximizing Margins with the Hinge Loss
A new loss function is proposed for neural networks on classification tasks which extends the hinge loss by assigning gradients to its critical points. We will show that for a linear classifier on linearly separable data with fixed step size, the margin of this modified hinge loss converges to the $\ell_2$ max-margin at the rate of $\mathcal{O}( 1/t )$. This rate is fast when compared with the $\mathcal{O}(1/\log t)$ rate of exponential losses such as the logistic loss. Furthermore, empirical results suggest that this increased convergence speed carries over to ReLU networks.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Spatially Decomposed Hinge Adversarial Loss by Local Gradient Amplifier
Generative Adversarial Networks (GANs) have achieved large attention and great success in various research areas, but it still suffers from training instability. Recently hinge adversarial loss for GAN is proposed that i…
On the implicit minimization of alternative loss functions when training deep networks
Understanding the implicit bias of optimization algorithms is important in order to improve generalization of neural networks. One approach to try to exploit such understanding would be to then make the bias explicit in …
Inductive BiasOn the Dual Formulation of Boosting Algorithms
We study boosting algorithms from a new perspective. We show that the Lagrange dual problems of AdaBoost, LogitBoost and soft-margin LPBoost with generalized hinge loss are all entropy maximization problems. By looking a…
Xtreme Margin: A Tunable Loss Function for Binary Classification Problems
Loss functions drive the optimization of machine learning algorithms. The choice of a loss function can have a significant impact on the training of a model, and how the model learns the data. Binary classification is on…
Binary ClassificationMMA Training: Direct Input Space Margin Maximization through Adversarial Training
We study adversarial robustness of neural networks from a margin maximization perspective, where margins are defined as the distances from inputs to a classifier's decision boundary. Our study shows that maximizing margi…
Adversarial DefenseAdversarial Robustness