SimFBO: Towards Simple, Flexible and Communication-efficient Federated Bilevel Learning
Federated bilevel optimization (FBO) has shown great potential recently in machine learning and edge computing due to the emerging nested optimization structure in meta-learning, fine-tuning, hyperparameter tuning, etc. However, existing FBO algorithms often involve complicated computations and require multiple sub-loops per iteration, each of which contains a number of communication rounds. In this paper, we propose a simple and flexible FBO framework named SimFBO, which is easy to implement without sub-loops, and includes a generalized server-side aggregation and update for improving communication efficiency. We further propose System-level heterogeneity robust FBO (ShroFBO) as a variant of SimFBO with stronger resilience to heterogeneous local computation. We show that SimFBO and ShroFBO provably achieve a linear convergence speedup with partial client participation and client sampling without replacement, as well as improved sample and communication complexities. Experiments demonstrate the effectiveness of the proposed methods over existing FBO algorithms.
Code (0)
등록된 구현이 없습니다.
Tasks
Bilevel OptimizationEdge-computingMeta-LearningSimilar Papers 제목 키워드 기반
Communication-Efficient Federated Hypergradient Computation via Aggregated Iterative Differentiation
Federated bilevel optimization has attracted increasing attention due to emerging machine learning and communication applications. The biggest challenge lies in computing the gradient of the upper-level objective functio…
Bilevel OptimizationCommunication-Efficient Federated Bilevel Optimization with Local and Global Lower Level Problems
Bilevel Optimization has witnessed notable progress recently with new emerging efficient algorithms. However, its application in the Federated Learning setting remains relatively underexplored, and the impact of Federate…
Bilevel OptimizationFederated LearningRepresentation LearningCommunication-Efficient Federated Bilevel Optimization with Global and Local Lower Level Problems
Bilevel Optimization has witnessed notable progress recently with new emerging efficient algorithms. However, its application in the Federated Learning setting remains relatively underexplored, and the impact of Federate…
Communication-Efficient Robust Federated Learning with Noisy Labels
Federated learning (FL) is a promising privacy-preserving machine learning paradigm over distributed located data. In FL, the data is kept locally by each user. This protects the user privacy, but also makes the server d…
Bilevel OptimizationFederated LearningLearning with noisy labelsPrivacy PreservingAchieving Linear Speedup in Non-IID Federated Bilevel Learning
Federated bilevel optimization has received increasing attention in various emerging machine learning and communication applications. Recently, several Hessian-vector-based algorithms have been proposed to solve the fede…
Bilevel OptimizationFederated Learning