On the Convergence of Distributed Stochastic Bilevel Optimization Algorithms over a Network
Bilevel optimization has been applied to a wide variety of machine learning models, and numerous stochastic bilevel optimization algorithms have been developed in recent years. However, most existing algorithms restrict their focus on the single-machine setting so that they are incapable of handling the distributed data. To address this issue, under the setting where all participants compose a network and perform peer-to-peer communication in this network, we developed two novel decentralized stochastic bilevel optimization algorithms based on the gradient tracking communication mechanism and two different gradient estimators. Additionally, we established their convergence rates for nonconvex-strongly-convex problems with novel theoretical analysis strategies. To our knowledge, this is the first work achieving these theoretical results. Finally, we applied our algorithms to practical machine learning models, and the experimental results confirmed the efficacy of our algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningBilevel OptimizationDistributed OptimizationSimilar Papers 제목 키워드 기반
On the Communication Complexity of Decentralized Bilevel Optimization
Stochastic bilevel optimization finds widespread applications in machine learning, including meta-learning, hyperparameter optimization, and neural architecture search. To extend stochastic bilevel optimization to distri…
Bilevel OptimizationHyperparameter OptimizationMeta-LearningNeural Architecture SearchLocal Stochastic Bilevel Optimization with Momentum-Based Variance Reduction
Bilevel Optimization has witnessed notable progress recently with new emerging efficient algorithms and has been applied to many machine learning tasks such as data cleaning, few-shot learning, and neural architecture se…
BIG-bench Machine LearningBilevel OptimizationFederated LearningFew-Shot Learning+1Decentralized Stochastic Bilevel Optimization with Improved per-Iteration Complexity
Bilevel optimization recently has received tremendous attention due to its great success in solving important machine learning problems like meta learning, reinforcement learning, and hyperparameter optimization. Extendi…
Bilevel OptimizationHyperparameter OptimizationMeta-LearningStochastic OptimizationBilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis
Bilevel optimization has become a powerful framework in various machine learning applications including meta-learning, hyperparameter optimization, and network architecture search. There are generally two classes of bile…
BIG-bench Machine LearningBilevel OptimizationHyperparameter OptimizationMeta-LearningAsynchronous Distributed Bilevel Optimization
Bilevel optimization plays an essential role in many machine learning tasks, ranging from hyperparameter optimization to meta-learning. Existing studies on bilevel optimization, however, focus on either centralized or sy…
Bilevel OptimizationHyperparameter OptimizationMeta-Learning