A deep cut into Split Federated Self-supervised Learning

2024-06-12 · Marcin Przewięźlikowski, Marcin Osial, Bartosz Zieliński, Marek Śmieja

Collaborative self-supervised learning has recently become feasible in highly distributed environments by dividing the network layers between client devices and a central server. However, state-of-the-art methods, such as MocoSFL, are optimized for network division at the initial layers, which decreases the protection of the client data and increases communication overhead. In this paper, we demonstrate that splitting depth is crucial for maintaining privacy and communication efficiency in distributed training. We also show that MocoSFL suffers from a catastrophic quality deterioration for the minimal communication overhead. As a remedy, we introduce Momentum-Aligned contrastive Split Federated Learning (MonAcoSFL), which aligns online and momentum client models during training procedure. Consequently, we achieve state-of-the-art accuracy while significantly reducing the communication overhead, making MonAcoSFL more practical in real-world scenarios.

📄 PDF Abstract BibTeX arXiv:2406.08267

Code (1)

gmum/monacosfl 공식 구현 pytorch

Tasks

Federated Learning Self-Supervised Learning

VFed-SSD: Towards Practical Vertical Federated Advertising

2022-05-31 · Wenjie Li, Qiaolin Xia, Junfeng Deng, Hao Cheng 외

As an emerging secure learning paradigm in lever-aging cross-agency private data, vertical federatedlearning (VFL) is expected to improve advertising models by enabling the joint learning of complementary user attributes…

Federated Learning Knowledge Distillation Transfer Learning

Federated Self-Supervised Learning of Monocular Depth Estimators for Autonomous Vehicles

2023-10-07 · Elton F. de S. Soares, Carlos Alberto V. Campos

Image-based depth estimation has gained significant attention in recent research on computer vision for autonomous vehicles in intelligent transportation systems. This focus stems from its cost-effectiveness and wide ran…

Autonomous Vehicles Depth Estimation Federated Learning Monocular Depth Estimation+3

FedNST: Federated Noisy Student Training for Automatic Speech Recognition

2022-06-06 · Haaris Mehmood, Agnieszka Dobrowolska, Karthikeyan Saravanan, Mete Ozay

Federated Learning (FL) enables training state-of-the-art Automatic Speech Recognition (ASR) models on user devices (clients) in distributed systems, hence preventing transmission of raw user data to a central server. A …

Automatic Speech Recognition Automatic Speech Recognition (ASR)Federated Learning Self-Supervised Learning+2

Federated Self-Supervised Contrastive Learning and Masked Autoencoder for Dermatological Disease Diagnosis

2022-08-24 · Yawen Wu, Dewen Zeng, Zhepeng Wang, Yi Sheng 외

In dermatological disease diagnosis, the private data collected by mobile dermatology assistants exist on distributed mobile devices of patients. Federated learning (FL) can use decentralized data to train models while k…

Contrastive Learning Federated Learning Self-Supervised Learning

Combined Federated and Split Learning in Edge Computing for Ubiquitous Intelligence in Internet of Things: State of the Art and Future Directions

2022-07-20 · Qiang Duan, Shijing Hu, Ruijun Deng, Zhihui Lu

Federated learning (FL) and split learning (SL) are two emerging collaborative learning methods that may greatly facilitate ubiquitous intelligence in Internet of Things (IoT). Federated learning enables machine learning…

Edge-computing Federated Learning