Regularization Cocktails
The regularization of prediction models is arguably the most crucial ingredient that allows Machine Learning solutions to generalize well on unseen data. Several types of regularization are popular in the Deep Learning community (e.g., weight decay, drop-out, early stopping, etc.), but so far these are selected on an adhoc basis, and there is no systematic study as to how different regularizers should be combined into the best "cocktail". In this paper, we fill this gap, by considering cocktails of 13 different regularization methods and framing the question of how to best combine them as a standard hyperparameter optimization problem. We perform a large-scale empirical study on 42 datasets, concluding that regularization cocktails substantially outperform individual regularization methods, even if the hyperparameters of the latter are carefully tuned; the optimal regularization cocktail depends on the dataset; and that regularization cocktails yield a higher gain on small datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Hyperparameter OptimizationSimilar Papers 제목 키워드 기반
An Iterative Algorithm for Regularized Non-negative Matrix Factorizations
We generalize the non-negative matrix factorization algorithm of Lee and Seung to accept a weighted norm, and to support ridge and Lasso regularization. We recast the Lee and Seung multiplicative update as an additive up…
Accelerated Discovery of Cryoprotectant Cocktails via Multi-Objective Bayesian Optimization
Designing cryoprotectant agent (CPA) cocktails for vitrification is challenging because formulations must be concentrated enough to suppress ice formation yet non-toxic enough to preserve cell viability. This tradeoff cr…
Well-tuned Simple Nets Excel on Tabular Datasets
Tabular datasets are the last "unconquered castle" for deep learning, with traditional ML methods like Gradient-Boosted Decision Trees still performing strongly even against recent specialized neural architectures. In th…
Inequality's Economic and Social Roots: the Role of Social Networks and Homophily
I discuss economic and social sources of inequality and elaborate on the role of social networks in inequality, economic immobility, and economic inefficiencies. The lens of social networks clarifies how the entanglement…
Open-Set Recognition of Breast Cancer Treatments
Open-set recognition generalizes a classification task by classifying test samples as one of the known classes from training or "unknown." As novel cancer drug cocktails with improved treatment are continually discovered…
Open Set LearningRobust classification