FedMAE: Federated Self-Supervised Learning with One-Block Masked Auto-Encoder
Latest federated learning (FL) methods started to focus on how to use unlabeled data in clients for training due to users' privacy concerns, high labeling costs, or lack of expertise. However, current Federated Semi-Supervised/Self-Supervised Learning (FSSL) approaches fail to learn large-scale images because of the limited computing resources of local clients. In this paper, we introduce a new framework FedMAE, which stands for Federated Masked AutoEncoder, to address the problem of how to utilize unlabeled large-scale images for FL. Specifically, FedMAE can pre-train one-block Masked AutoEncoder (MAE) using large images in lightweight client devices, and then cascades multiple pre-trained one-block MAEs in the server to build a multi-block ViT backbone for downstream tasks. Theoretical analysis and experimental results on image reconstruction and classification show that our FedMAE achieves superior performance compared to the state-of-the-art FSSL methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningImage ReconstructionSelf-Supervised LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Federated Self-Supervised Contrastive Learning and Masked Autoencoder for Dermatological Disease Diagnosis
In dermatological disease diagnosis, the private data collected by mobile dermatology assistants exist on distributed mobile devices of patients. Federated learning (FL) can use decentralized data to train models while k…
Contrastive LearningFederated LearningSelf-Supervised LearningEntropy Reveals Block Importance in Masked Self-Supervised Vision Transformers
Masked self-supervised vision transformers have become a dominant pretraining paradigm, yet their substantial model size poses significant challenges for resource-constrained deployment and efficient transfer learning. A…
Transfer LearningModel CompressionSelf-Supervised Predictive Convolutional Attentive Block for Anomaly Detection
Anomaly detection is commonly pursued as a one-class classification problem, where models can only learn from normal training samples, while being evaluated on both normal and abnormal test samples. Among the successful …
Anomaly DetectionOne-Class ClassificationSelf-Supervised Masked Convolutional Transformer Block for Anomaly Detection
Anomaly detection has recently gained increasing attention in the field of computer vision, likely due to its broad set of applications ranging from product fault detection on industrial production lines and impending ev…
Anomaly DetectionEvent DetectionFault DetectionOne-Class ClassificationDepth-Wise Representation Development Under Blockwise Self-Supervised Learning for Video Vision Transformers
End-to-end backpropagation couples all layers through a global error signal, enabling coordinated learning but requiring long-range credit assignment. Motivated by recent progress in blockwise self-supervised learning (B…
Self-Supervised Learning