Learning Effective Loss Functions Efficiently
We consider the problem of learning a loss function which, when minimized over a training dataset, yields a model that approximately minimizes a validation error metric. Though learning an optimal loss function is NP-hard, we present an anytime algorithm that is asymptotically optimal in the worst case, and is provably efficient in an idealized "easy" case. Experimentally, we show that this algorithm can be used to tune loss function hyperparameters orders of magnitude faster than state-of-the-art alternatives. We also show that our algorithm can be used to learn novel and effective loss functions on-the-fly during training.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Boosting in the presence of outliers: adaptive classification with non-convex loss functions
This paper examines the role and efficiency of the non-convex loss functions for binary classification problems. In particular, we investigate how to design a simple and effective boosting algorithm that is robust to the…
Binary ClassificationGeneral ClassificationMeta-learning Loss Functions of Parametric Partial Differential Equations Using Physics-Informed Neural Networks
This paper proposes a new way to learn Physics-Informed Neural Network loss functions using Generalized Additive Models. We apply our method by meta-learning parametric partial differential equations, PDEs, on Burger's a…
Additive modelsMeta-LearningPersistent Reductions in Regularized Loss Minimization for Variable Selection
In the context of regularized loss minimization with polyhedral gauges, we show that for a broad class of loss functions (possibly non-smooth and non-convex) and under a simple geometric condition on the input data it is…
Variable SelectionData Augmentation for Text Generation Without Any Augmented Data
Data augmentation is an effective way to improve the performance of many neural text generation models. However, current data augmentation methods need to define or choose proper data mapping functions that map the origi…
Data AugmentationText GenerationAn empirical analysis of the optimization of deep network loss surfaces
The success of deep neural networks hinges on our ability to accurately and efficiently optimize high-dimensional, non-convex functions. In this paper, we empirically investigate the loss functions of state-of-the-art ne…
Stochastic Optimization