Asynchronous Distributed Bilevel Optimization

2022-12-20 · Yang Jiao, Kai Yang, Tiancheng Wu, Dongjin Song, Chengtao Jian

Bilevel optimization plays an essential role in many machine learning tasks, ranging from hyperparameter optimization to meta-learning. Existing studies on bilevel optimization, however, focus on either centralized or synchronous distributed setting. The centralized bilevel optimization approaches require collecting massive amount of data to a single server, which inevitably incur significant communication expenses and may give rise to data privacy risks. Synchronous distributed bilevel optimization algorithms, on the other hand, often face the straggler problem and will immediately stop working if a few workers fail to respond. As a remedy, we propose Asynchronous Distributed Bilevel Optimization (ADBO) algorithm. The proposed ADBO can tackle bilevel optimization problems with both nonconvex upper-level and lower-level objective functions, and its convergence is theoretically guaranteed. Furthermore, it is revealed through theoretic analysis that the iteration complexity of ADBO to obtain the $\epsilon$-stationary point is upper bounded by $\mathcal{O}(\frac{1}{{{\epsilon ^2}}})$. Thorough empirical studies on public datasets have been conducted to elucidate the effectiveness and efficiency of the proposed ADBO.

📄 PDF Abstract BibTeX arXiv:2212.10048

Code (1)

iclr23submission6251/adbo 공식 구현 pytorch

Tasks

Bilevel Optimization Hyperparameter Optimization Meta-Learning

Methods 이 논문이 사용한 방법론

fail 설명 없음

AsynDBT: Asynchronous Distributed Bilevel Tuning for efficient In-Context Learning with Large Language Models

2026-02-06 · Hui Ma, Shaoyu Dou, Ya Liu, Fei Xing 외 arxiv

With the rapid development of large language models (LLMs), an increasing number of applications leverage cloud-based LLM APIs to reduce usage costs. However, since cloud-based models' parameters and gradients are agnost…

Federated Learning

Fast Adaptive Federated Bilevel Optimization

2022-11-02 · Feihu Huang

Bilevel optimization is a popular hierarchical model in machine learning, and has been widely applied to many machine learning tasks such as meta learning, hyperparameter learning and policy optimization. Although many b…

Bilevel Optimization Distributed Optimization Meta-Learning Representation Learning

Argus: Federated Non-convex Bilevel Learning over 6G Space-Air-Ground Integrated Network

2025-05-14 · Ya Liu, Kai Yang, Yu Zhu, Keying Yang 외

The space-air-ground integrated network (SAGIN) has recently emerged as a core element in the 6G networks. However, traditional centralized and synchronous optimization algorithms are unsuitable for SAGIN due to infrastr…

Decentralized Gossip-Based Stochastic Bilevel Optimization over Communication Networks

2022-06-22 · Shuoguang Yang, Xuezhou Zhang, Mengdi Wang

Bilevel optimization have gained growing interests, with numerous applications found in meta learning, minimax games, reinforcement learning, and nested composition optimization. This paper studies the problem of distrib…

Bilevel Optimization Federated Learning Meta-Learning reinforcement-learning+2

On the Convergence of Distributed Stochastic Bilevel Optimization Algorithms over a Network

2022-06-30 · Hongchang Gao, Bin Gu, My T. Thai

Bilevel optimization has been applied to a wide variety of machine learning models, and numerous stochastic bilevel optimization algorithms have been developed in recent years. However, most existing algorithms restrict …

BIG-bench Machine Learning Bilevel Optimization Distributed Optimization