Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data

2025-04-20 · Yuting He, Yiqiang Chen, Xiaodong Yang, Hanchao Yu, Yi-Hua Huang, Yang Gu

Federated learning (FL) enables multiple clients to collaboratively train a global model while keeping local data decentralized. Data heterogeneity (non-IID) across clients has imposed significant challenges to FL, which makes local models re-optimize towards their own local optima and forget the global knowledge, resulting in performance degradation and convergence slowdown. Many existing works have attempted to address the non-IID issue by adding an extra global-model-based regularizing item to the local training but without an adaption scheme, which is not efficient enough to achieve high performance with deep learning models. In this paper, we propose a Selective Self-Distillation method for Federated learning (FedSSD), which imposes adaptive constraints on the local updates by self-distilling the global model's knowledge and selectively weighting it by evaluating the credibility at both the class and sample level. The convergence guarantee of FedSSD is theoretically analyzed and extensive experiments are conducted on three public benchmark datasets, which demonstrates that FedSSD achieves better generalization and robustness in fewer communication rounds, compared with other state-of-the-art FL methods.

📄 PDF Abstract BibTeX arXiv:2504.14694

Code (0)

등록된 구현이 없습니다.

Tasks

Federated Learning

FedSDR: Federated Self-Distillation with Rectification

2026-05-18 · Ziheng Ren, Zhanming Shen, Hao Wang, Ning Liu 외 arxiv

Federated fine-tuning of Large Language Models faces severe statistical heterogeneity. However, existing model-level defenses often overlook the root cause: intrinsic data distribution mismatches. In this work, we first …

Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher

2023-04-04 · Jiawei Shao, Fangzhao Wu, Jun Zhang

While federated learning is promising for privacy-preserving collaborative learning without revealing local data, it remains vulnerable to white-box attacks and struggles to adapt to heterogeneous clients. Federated dist…

Federated Learning Knowledge Distillation Privacy Preserving Transfer Learning

Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarization

2023-08-07 · Xiachong Feng, Xiaocheng Feng, Xiyuan Du, Min-Yen Kan 외

Meeting summarization has emerged as a promising technique for providing users with condensed summaries. However, existing work has focused on training models on centralized data, neglecting real-world scenarios where me…

Federated Learning Knowledge Distillation Meeting Summarization

Oblivionis: A Lightweight Learning and Unlearning Framework for Federated Large Language Models

2025-08-12 · Fuyao Zhang, Xinyu Yan, Tiantong Wu, Wenjie Li 외 arxiv

Large Language Models (LLMs) increasingly leverage Federated Learning (FL) to utilize private, task-specific datasets for fine-tuning while preserving data privacy. However, while federated LLM frameworks effectively ena…

Federated Learning

FedBiCross: Personalized One-Shot Federated Learning on Medical Images

2026-01-05 · Yuexuan Xia, Yinghao Zhang, Yalin Liu, Hong-Ning Dai 외 arxiv

Data-free knowledge distillation-based one-shot federated learning (OSFL) trains a model in a single communication round without sharing raw data, making OSFL attractive for privacy-sensitive medical applications. Howeve…

Data-free Knowledge Distillation Federated Learning