Relaxed Contrastive Learning for Federated Learning
We propose a novel contrastive learning framework to effectively address the challenges of data heterogeneity in federated learning. We first analyze the inconsistency of gradient updates across clients during local training and establish its dependence on the distribution of feature representations, leading to the derivation of the supervised contrastive learning (SCL) objective to mitigate local deviations. In addition, we show that a na\"ive adoption of SCL in federated learning leads to representation collapse, resulting in slow convergence and limited performance gains. To address this issue, we introduce a relaxed contrastive learning loss that imposes a divergence penalty on excessively similar sample pairs within each class. This strategy prevents collapsed representations and enhances feature transferability, facilitating collaborative training and leading to significant performance improvements. Our framework outperforms all existing federated learning approaches by huge margins on the standard benchmarks through extensive experimental results.
Code (1)
Tasks
Contrastive LearningFederated LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Practical Validity Conditions for Byzantine-Tolerant Federated Learning
Robust aggregation is the core operation in Byzantine-tolerant federated learning. To ensure the quality of aggregation independently of data distribution or attacks, validity conditions are needed. They provide geometri…
Federated LearningUnderstanding How Consistency Works in Federated Learning via Stage-wise Relaxed Initialization
Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the heterogeneous dataset. Previous works have i…
Federated LearningDecoupled Contrastive Learning for Federated Learning
Federated learning is a distributed machine learning paradigm that allows multiple participants to train a shared model by exchanging model updates instead of their raw data. However, its performance is degraded compared…
Contrastive LearningFederated LearningModel-Contrastive Federated Learning
Federated learning enables multiple parties to collaboratively train a machine learning model without communicating their local data. A key challenge in federated learning is to handle the heterogeneity of local data dis…
Contrastive LearningFederated Learningimage-classificationImage Classification+1PLReMix: Combating Noisy Labels with Pseudo-Label Relaxed Contrastive Representation Learning
Recently, the usage of Contrastive Representation Learning (CRL) as a pre-training technique improves the performance of learning with noisy labels (LNL) methods. However, instead of pre-training, when trivially combinin…
Learning with noisy labelsPseudo LabelRepresentation Learning