Multi-Level Branched Regularization for Federated Learning
A critical challenge of federated learning is data heterogeneity and imbalance across clients, which leads to inconsistency between local networks and unstable convergence of global models. To alleviate the limitations, we propose a novel architectural regularization technique that constructs multiple auxiliary branches in each local model by grafting local and global subnetworks at several different levels and that learns the representations of the main pathway in the local model congruent to the auxiliary hybrid pathways via online knowledge distillation. The proposed technique is effective to robustify the global model even in the non-iid setting and is applicable to various federated learning frameworks conveniently without incurring extra communication costs. We perform comprehensive empirical studies and demonstrate remarkable performance gains in terms of accuracy and efficiency compared to existing methods. The source code is available at our project page.
Code (2)
Tasks
Federated LearningKnowledge DistillationSimilar Papers 제목 키워드 기반
Dynamic Heterogeneous Federated Learning with Multi-Level Prototypes
Federated learning shows promise as a privacy-preserving collaborative learning technique. Existing heterogeneous federated learning mainly focuses on skewing the label distribution across clients. However, most approach…
Federated LearningPrivacy PreservingMimicking Ensemble Learning with Deep Branched Networks
This paper proposes a branched residual network for image classification. It is known that high-level features of deep neural network are more representative than lower-level features. By sharing the low-level features, …
ClassificationEnsemble LearningGeneral Classificationimage-classification+1Branched Schrödinger Bridge Matching
Predicting the intermediate trajectories between an initial and target distribution is a central problem in generative modeling. Existing approaches, such as flow matching and Schr\"odinger Bridge Matching, effectively l…
The new face of multifractality: Multi-branchedness and the phase transitions in time series of mean inter-event times
Empirical time series of inter-event or waiting times are investigated using a modified Multifractal Detrended Fluctuation Analysis operating on fluctuations of mean detrended dynamics. The core of the extended multifrac…
Time SeriesTime Series AnalysisDifferentially Private Federated Learning with Local Regularization and Sparsification
User-level differential privacy (DP) provides certifiable privacy guarantees to the information that is specific to any user's data in federated learning. Existing methods that ensure user-level DP come at the cost of se…
Federated Learning