Federated Training of Dual Encoding Models on Small Non-IID Client Datasets
Dual encoding models that encode a pair of inputs are widely used for representation learning. Many approaches train dual encoding models by maximizing agreement between pairs of encodings on centralized training data. However, in many scenarios, datasets are inherently decentralized across many clients (user devices or organizations) due to privacy concerns, motivating federated learning. In this work, we focus on federated training of dual encoding models on decentralized data composed of many small, non-IID (independent and identically distributed) client datasets. We show that existing approaches that work well in centralized settings perform poorly when naively adapted to this setting using federated averaging. We observe that, we can simulate large-batch loss computation on individual clients for loss functions that are based on encoding statistics. Based on this insight, we propose a novel federated training approach, Distributed Cross Correlation Optimization (DCCO), which trains dual encoding models using encoding statistics aggregated across clients, without sharing individual data samples. Our experimental results on two datasets demonstrate that the proposed DCCO approach outperforms federated variants of existing approaches by a large margin.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningRepresentation LearningSimilar Papers 제목 키워드 기반
Controlling Participation in Federated Learning with Feedback
We address the problem of client participation in federated learning, where traditional methods typically rely on a random selection of a small subset of clients for each training round. In contrast, we propose FedBack, …
Computational EfficiencyFederated Learningimage-classificationImage ClassificationFedCVESA: Taking Away Training Data in Federated Learning via Correlation Value Encoding and Segmented Aggregation
Federated learning (FL) avoids explicit data exposure by keeping raw data on local clients, yet privacy risks remain in the training process and the learned model itself. Recently, centralized Taking Away Training Data (…
Federated LearningToward Understanding the Influence of Individual Clients in Federated Learning
Federated learning allows mobile clients to jointly train a global model without sending their private data to a central server. Extensive works have studied the performance guarantee of the global model, however, it is …
Federated LearningFedRE: A Representation Entanglement Framework for Model-Heterogeneous Federated Learning
Federated learning (FL) enables collaborative training across clients while preserving privacy. While most existing FL methods assume homogeneous model architectures, client heterogeneity in both data and resources makes…
Federated LearningCombating Data Imbalances in Federated Semi-supervised Learning with Dual Regulators
Federated learning has become a popular method to learn from decentralized heterogeneous data. Federated semi-supervised learning (FSSL) emerges to train models from a small fraction of labeled data due to label scarcity…
Federated Learning