Min-Max Bilevel Multi-objective Optimization with Applications in Machine Learning
We consider a generic min-max multi-objective bilevel optimization problem with applications in robust machine learning such as representation learning and hyperparameter optimization. We design MORBiT, a novel single-loop gradient descent-ascent bilevel optimization algorithm, to solve the generic problem and present a novel analysis showing that MORBiT converges to the first-order stationary point at a rate of $\widetilde{\mathcal{O}}(n^{1/2} K^{-2/5})$ for a class of weakly convex problems with $n$ objectives upon $K$ iterations of the algorithm. Our analysis utilizes novel results to handle the non-smooth min-max multi-objective setup and to obtain a sublinear dependence in the number of objectives $n$. Experimental results on robust representation learning and robust hyperparameter optimization showcase (i) the advantages of considering the min-max multi-objective setup, and (ii) convergence properties of the proposed MORBiT. Our code is at https://github.com/minimario/MORBiT.
Code (1)
Tasks
BIG-bench Machine LearningBilevel OptimizationHyperparameter OptimizationMeta-LearningMulti-Task LearningRepresentation LearningSimilar Papers 제목 키워드 기반
A Single-Timescale Method for Stochastic Bilevel Optimization
Stochastic bilevel optimization generalizes the classic stochastic optimization from the minimization of a single objective to the minimization of an objective function that depends the solution of another optimization p…
Bilevel OptimizationMeta-LearningStochastic OptimizationA Lower Bound and a Near-Optimal Algorithm for Bilevel Empirical Risk Minimization
Bilevel optimization problems, which are problems where two optimization problems are nested, have more and more applications in machine learning. In many practical cases, the upper and the lower objectives correspond to…
Bilevel OptimizationEfficient Hessian-Free Methods for Multi-Objective Bilevel Optimization with Nonconvex Lower Level
Multi-objective bilevel optimization has wide applications in the AI area such as automated learning and multi-task meta-learning. Although recently some works have been begun to study the multi-objective bilevel optimiz…
Neural Architecture SearchBilevel OptimizationAn Alternating Optimization Method for Bilevel Problems under the Polyak-Łojasiewicz Condition
Bilevel optimization has recently regained interest owing to its applications in emerging machine learning fields such as hyperparameter optimization, meta-learning, and reinforcement learning. Recent results have shown …
On Penalty-based Bilevel Gradient Descent Method
Bilevel optimization enjoys a wide range of applications in emerging machine learning and signal processing problems such as hyper-parameter optimization, image reconstruction, meta-learning, adversarial training, and re…
Bilevel OptimizationImage ReconstructionMeta-Learningreinforcement-learning+1