Clustered Sampling: Low-Variance and Improved Representativity for Clients Selection in Federated Learning
This work addresses the problem of optimizing communications between server and clients in federated learning (FL). Current sampling approaches in FL are either biased, or non optimal in terms of server-clients communications and training stability. To overcome this issue, we introduce \textit{clustered sampling} for clients selection. We prove that clustered sampling leads to better clients representatitivity and to reduced variance of the clients stochastic aggregation weights in FL. Compatibly with our theory, we provide two different clustering approaches enabling clients aggregation based on 1) sample size, and 2) models similarity. Through a series of experiments in non-iid and unbalanced scenarios, we demonstrate that model aggregation through clustered sampling consistently leads to better training convergence and variability when compared to standard sampling approaches. Our approach does not require any additional operation on the clients side, and can be seamlessly integrated in standard FL implementations. Finally, clustered sampling is compatible with existing methods and technologies for privacy enhancement, and for communication reduction through model compression.
Code (1)
Tasks
ClusteringFederated LearningModel CompressionSimilar Papers 제목 키워드 기반
Design-Based Multi-Way Clustering
This paper extends the design-based framework to settings with multi-way cluster dependence, and shows how multi-way clustering can be justified when clustered assignment and clustered sampling occurs on different dimens…
ClusteringvalidPrediction of microstructural representativity from a single image
In this study, we present a method for predicting the representativity of the phase fraction observed in a single image (2D or 3D) of a material. Traditional approaches often require large datasets and extensive statisti…
PredictionFind Your Optimal Assignments On-the-fly: A Holistic Framework for Clustered Federated Learning
Federated Learning (FL) is an emerging distributed machine learning approach that preserves client privacy by storing data on edge devices. However, data heterogeneity among clients presents challenges in training models…
ClusteringFederated LearningTowards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matching
In federated learning (FL), clients may have diverse objectives, and merging all clients' knowledge into one global model will cause negative transfer to local performance. Thus, clustered FL is proposed to group similar…
Federated LearningInteraction-Aware Gaussian Weighting for Clustered Federated Learning
Federated Learning (FL) emerged as a decentralized paradigm to train models while preserving privacy. However, conventional FL struggles with data heterogeneity and class imbalance, which degrade model performance. Clust…
ClusteringFederated Learning