An Adaptively Inexact Method for Bilevel Learning Using Primal-Dual Style Differentiation
We consider a bilevel learning framework for learning linear operators. In this framework, the learnable parameters are optimized via a loss function that also depends on the minimizer of a convex optimization problem (denoted lower-level problem). We utilize an iterative algorithm called `piggyback' to compute the gradient of the loss and minimizer of the lower-level problem. Given that the lower-level problem is solved numerically, the loss function and thus its gradient can only be computed inexactly. To estimate the accuracy of the computed hypergradient, we derive an a-posteriori error bound, which provides guides for setting the tolerance for the lower-level problem, as well as the piggyback algorithm. To efficiently solve the upper-level optimization, we also propose an adaptive method for choosing a suitable step-size. To illustrate the proposed method, we consider a few learned regularizer problems, such as training an input-convex neural network.
Code (1)
Similar Papers 제목 키워드 기반
Inexact Primal-Dual Gradient Projection Methods for Nonlinear Optimization on Convex Set
In this paper, we propose a novel primal-dual inexact gradient projection method for nonlinear optimization problems with convex-set constraint. This method only needs inexact computation of the projections onto the conv…
Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems
Two-level stochastic optimization formulations have become instrumental in a number of machine learning contexts such as continual learning, neural architecture search, adversarial learning, and hyperparameter tuning. Pr…
BIG-bench Machine LearningBilevel OptimizationContinual LearningNeural Architecture Search+1A Primal-Dual Approach to Bilevel Optimization with Multiple Inner Minima
Bilevel optimization has found extensive applications in modern machine learning problems such as hyperparameter optimization, neural architecture search, meta-learning, etc. While bilevel problems with a unique inner mi…
Bilevel OptimizationHyperparameter OptimizationMeta-LearningNeural Architecture SearchAn adaptively inexact first-order method for bilevel optimization with application to hyperparameter learning
Various tasks in data science are modeled utilizing the variational regularization approach, where manually selecting regularization parameters presents a challenge. The difficulty gets exacerbated when employing regular…
Bilevel OptimizationDenoisingBilevel Learning with Inexact Stochastic Gradients
Bilevel learning has gained prominence in machine learning, inverse problems, and imaging applications, including hyperparameter optimization, learning data-adaptive regularizers, and optimizing forward operators. The la…
Bilevel OptimizationDeblurringDenoisingHyperparameter Optimization+2