Gradient-based bilevel optimization for multi-penalty Ridge regression through matrix differential calculus
Common regularization algorithms for linear regression, such as LASSO and Ridge regression, rely on a regularization hyperparameter that balances the tradeoff between minimizing the fitting error and the norm of the learned model coefficients. As this hyperparameter is scalar, it can be easily selected via random or grid search optimizing a cross-validation criterion. However, using a scalar hyperparameter limits the algorithm's flexibility and potential for better generalization. In this paper, we address the problem of linear regression with l2-regularization, where a different regularization hyperparameter is associated with each input variable. We optimize these hyperparameters using a gradient-based approach, wherein the gradient of a cross-validation criterion with respect to the regularization hyperparameters is computed analytically through matrix differential calculus. Additionally, we introduce two strategies tailored for sparse model learning problems aiming at reducing the risk of overfitting to the validation data. Numerical examples demonstrate that our multi-hyperparameter regularization approach outperforms LASSO, Ridge, and Elastic Net regression. Moreover, the analytical computation of the gradient proves to be more efficient in terms of computational time compared to automatic differentiation, especially when handling a large number of input variables. Application to the identification of over-parameterized Linear Parameter-Varying models is also presented.
Code (1)
Tasks
Bilevel OptimizationL2 RegularizationregressionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On Penalty-based Bilevel Gradient Descent Method
Bilevel optimization enjoys a wide range of applications in emerging machine learning and signal processing problems such as hyper-parameter optimization, image reconstruction, meta-learning, adversarial training, and re…
Bilevel OptimizationImage ReconstructionMeta-Learningreinforcement-learning+1A Penalty-Based Method for Communication-Efficient Decentralized Bilevel Programming
Bilevel programming has recently received attention in the literature due to its wide range of applications, including reinforcement learning and hyper-parameter optimization. However, it is widely assumed that the under…
Bilevel OptimizationFederated LearningSample Complexity Analysis for Constrained Bilevel Reinforcement Learning
Several important problem settings within the literature of reinforcement learning (RL), such as meta-learning, hierarchical learning, and RL from human feedback (RL-HF), can be modelled as bilevel RL problems. A lot has…
Reinforcement LearningPrincipled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF
Bilevel optimization has been recently applied to many machine learning tasks. However, their applications have been restricted to the supervised learning setting, where static objective functions with benign structures …
Bilevel Optimizationreinforcement-learningReinforcement LearningReinforcement Learning (RL)Penalty Method for Inversion-Free Deep Bilevel Optimization
Solving a bilevel optimization problem is at the core of several machine learning problems such as hyperparameter tuning, data denoising, meta- and few-shot learning, and training-data poisoning. Different from simultane…
Bilevel OptimizationData PoisoningDenoisingFew-Shot Learning