Optimizing for Generalization in Machine Learning with Cross-Validation Gradients
Cross-validation is the workhorse of modern applied statistics and machine learning, as it provides a principled framework for selecting the model that maximizes generalization performance. In this paper, we show that the cross-validation risk is differentiable with respect to the hyperparameters and training data for many common machine learning algorithms, including logistic regression, elastic-net regression, and support vector machines. Leveraging this property of differentiability, we propose a cross-validation gradient method (CVGM) for hyperparameter optimization. Our method enables efficient optimization in high-dimensional hyperparameter spaces of the cross-validation risk, the best surrogate of the true generalization ability of our learning algorithm.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningHyperparameter OptimizationregressionSimilar Papers 제목 키워드 기반
Optimizing for Generalization in Machine Learning with Cross-Validation Gradients
Cross-validation is the workhorse of modern applied statistics and machine learning, as it provides a principled framework for selecting the model that maximizes generalization performance. In this paper, we show that th…
BIG-bench Machine LearningHyperparameter OptimizationregressionThe Theory Behind Overfitting, Cross Validation, Regularization, Bagging, and Boosting: Tutorial
In this tutorial paper, we first define mean squared error, variance, covariance, and bias of both random variables and classification/predictor models. Then, we formulate the true and generalization errors of the model …
L2 RegularizationDeep Bilevel Learning
We present a novel regularization approach to train neural networks that enjoys better generalization and test error than standard stochastic gradient descent. Our approach is based on the principles of cross-validation,…
Bilevel OptimizationEarly Stopping without a Validation Set
Early stopping is a widely used technique to prevent poor generalization performance when training an over-expressive model by means of gradient-based optimization. To find a good point to halt the optimizer, a common pr…
regressionInformation-Theoretic Generalization Bounds for Stochastic Gradient Descent
We study the generalization properties of the popular stochastic optimization method known as stochastic gradient descent (SGD) for optimizing general non-convex loss functions. Our main contribution is providing upper b…
Generalization BoundsStochastic Optimization