Cross-feature Contrastive Loss for Decentralized Deep Learning on Heterogeneous Data

2023-10-24 · Sai Aparna Aketi, Kaushik Roy

The current state-of-the-art decentralized learning algorithms mostly assume the data distribution to be Independent and Identically Distributed (IID). However, in practical scenarios, the distributed datasets can have significantly heterogeneous data distributions across the agents. In this work, we present a novel approach for decentralized learning on heterogeneous data, where data-free knowledge distillation through contrastive loss on cross-features is utilized to improve performance. Cross-features for a pair of neighboring agents are the features (i.e., last hidden layer activations) obtained from the data of an agent with respect to the model parameters of the other agent. We demonstrate the effectiveness of the proposed technique through an exhaustive set of experiments on various Computer Vision datasets (CIFAR-10, CIFAR-100, Fashion MNIST, Imagenette, and ImageNet), model architectures, and network topologies. Our experiments show that the proposed method achieves superior performance (0.2-4% improvement in test accuracy) compared to other existing techniques for decentralized learning on heterogeneous data.

📄 PDF Abstract BibTeX arXiv:2310.15890

Code (1)

aparna-aketi/cross_feature_contrastive_loss 공식 구현 pytorch

Tasks

Data-free Knowledge Distillation Knowledge Distillation

Methods 이 논문이 사용한 방법론

SET Dynamic Sparse Training method where weight mask is updated randomly periodically

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

FedSSC: Shared Supervised-Contrastive Federated Learning

2023-01-14 · Sirui Hu, Ling Feng, Xiaohan Yang, Yongchao Chen

Federated learning is widely used to perform decentralized training of a global model on multiple devices while preserving the data privacy of each device. However, it suffers from heterogeneous local data on each traini…

Contrastive Learning Federated Learning

Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data

2026-07-02 · Xuanyu Chen, Nan Yang, Shuai Wang, Dong Yuan arxiv

Recent research has introduced distributed self-supervised learning (D-SSL) approaches to leverage vast amounts of unlabeled decentralized data. However, D-SSL faces the critical challenge of data heterogeneity, and ther…

Self-Supervised Learning Contrastive Learning Federated Learning

MAPL: Model Agnostic Peer-to-peer Learning

2024-03-28 · Sayak Mukherjee, Andrea Simonetto, Hadi Jamali-Rad

Effective collaboration among heterogeneous clients in a decentralized setting is a rather unexplored avenue in the literature. To structurally address this, we introduce Model Agnostic Peer-to-peer Learning (coined as M…

Graph Learning model Privacy Preserving

Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision?

2022-10-20 · Lirui Wang, Kaiqing Zhang, Yunzhu Li, Yonglong Tian 외

Decentralized learning has been advocated and widely deployed to make efficient use of distributed datasets, with an extensive focus on supervised learning (SL) problems. Unfortunately, the majority of real-world data ar…

Contrastive Learning Representation Learning Self-Supervised Learning

FedX: Unsupervised Federated Learning with Cross Knowledge Distillation

2022-07-19 · Sungwon Han, Sungwon Park, Fangzhao Wu, Sundong Kim 외

This paper presents FedX, an unsupervised federated learning framework. Our model learns unbiased representation from decentralized and heterogeneous local data. It employs a two-sided knowledge distillation with contras…

Contrastive Learning Federated Learning Knowledge Distillation