A Gradient Method for Multilevel Optimization
Although application examples of multilevel optimization have already been discussed since the 1990s, the development of solution methods was almost limited to bilevel cases due to the difficulty of the problem. In recent years, in machine learning, Franceschi et al. have proposed a method for solving bilevel optimization problems by replacing their lower-level problems with the $T$ steepest descent update equations with some prechosen iteration number $T$. In this paper, we have developed a gradient-based algorithm for multilevel optimization with $n$ levels based on their idea and proved that our reformulation asymptotically converges to the original multilevel problem. As far as we know, this is one of the first algorithms with some theoretical guarantee for multilevel optimization. Numerical experiments show that a trilevel hyperparameter learning model considering data poisoning produces more stable prediction results than an existing bilevel hyperparameter learning model in noisy data settings.
Code (0)
등록된 구현이 없습니다.
Tasks
Bilevel OptimizationData PoisoningSimilar Papers 제목 키워드 기반
Towards Differentiable Multilevel Optimization: A Gradient-Based Approach
Multilevel optimization has gained renewed interest in machine learning due to its promise in applications such as hyperparameter tuning and continual learning. However, existing methods struggle with the inherent diffic…
Continual LearningMultilevel Monte Carlo Variational Inference
We propose a variance reduction framework for variational inference using the Multilevel Monte Carlo (MLMC) method. Our framework is built on reparameterized gradient estimators and "recycles" parameters obtained from pa…
Stochastic OptimizationVariational InferenceA Multilevel Approach to Training
We propose a novel training method based on nonlinear multilevel minimization techniques, commonly used for solving discretized large scale partial differential equations. Our multilevel training method constructs a mult…
Efficient Debiased Evidence Estimation by Multilevel Monte Carlo Sampling
In this paper, we propose a new stochastic optimization algorithm for Bayesian inference based on multilevel Monte Carlo (MLMC) methods. In Bayesian statistics, biased estimators of the model evidence have been often use…
Bayesian InferenceStochastic OptimizationFurther analysis of multilevel Stein variational gradient descent with an application to the Bayesian inference of glacier ice models
Multilevel Stein variational gradient descent is a method for particle-based variational inference that leverages hierarchies of surrogate target distributions with varying costs and fidelity to computationally speed up …
Bayesian InferenceVariational Inference