Demystifying Hyperparameter Optimization in Federated Learning
Federated Learning (FL) is a new machine learning paradigm that enables training models collaboratively across clients without sharing private data. In FL, data is non-uniformly distributed among clients (i.e., data heterogeneity) and cannot be balanced nor monitored like in conventional ML. Such data heterogeneity and privacy requirements bring unique challenges for learning hyperparameter optimization as the training dynamics change across clients even within the same training round and they are difficult to measure due to privacy constraints. State-of-the-art frameworks in FL focus on developing better aggregation algorithms and policies with the aim of mitigating these challenges. However, almost all existing FL systems adopt a ``global'' tuning method that uses a single set of learning hyperparameters across all the clients, regardless of their underlying data distributions. Our study shows that such a widely adopted global tuning method is not suitable for FL due to its data heterogeneity-oblivious nature. We demonstrate that the data quantity and distribution of the clients have a significant impact on the choice of hyperparameters, making it necessary to have customized tuning for each client. Based on these observations, we propose a first of its kind heterogeneity-aware hyperparameter optimization methodology, FedTune, that adopts a proxy data based hyperparameter customization approach to address the privacy and tuning cost challenges. Together with a Bayesian strengthened tuner, the proposed customized tuning approach is effective, lightweight, and privacy preserving. Extensive evaluation demonstrates that FedTune can achieve up to 7/4/4/6% better accuracy than the widely adopted globally tuned method for popular FL benchmarks FEMNIST, Cifar100, Cifar10, and Fashion-MNIST respectively.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningHyperparameter OptimizationPrivacy PreservingSimilar Papers 제목 키워드 기반
Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
Tuning hyperparameters is a crucial but arduous part of the machine learning pipeline. Hyperparameter optimization is even more challenging in federated learning, where models are learned over a distributed network of he…
Federated LearningHyperparameter OptimizationNeural Architecture SearchEvaluation of Hyperparameter-Optimization Approaches in an Industrial Federated Learning System
Federated Learning (FL) decouples model training from the need for direct access to the data and allows organizations to collaborate with industry partners to reach a satisfying level of performance without sharing vulne…
Bayesian OptimizationFederated LearningHyperparameter OptimizationGeneralizable Hyperparameter Optimization for Federated Learning on Non-IID Cancer Images
Deep learning for cancer histopathology training conflicts with privacy constraints in clinical settings. Federated Learning (FL) mitigates this by keeping data local; however, its performance depends on hyperparameter c…
Hyperparameter OptimizationFederated LearningFederated Hypergradient Descent
In this work, we explore combining automatic hyperparameter tuning and optimization for federated learning (FL) in an online, one-shot procedure. We apply a principled approach on a method for adaptive client learning ra…
Federated LearningHyperparameter OptimizationAuto-FedRL: Federated Hyperparameter Optimization for Multi-institutional Medical Image Segmentation
Federated learning (FL) is a distributed machine learning technique that enables collaborative model training while avoiding explicit data sharing. The inherent privacy-preserving property of FL algorithms makes them esp…
Federated LearningHyperparameter OptimizationImage SegmentationLesion Segmentation+6