Tailored Federated Learning: Leveraging Direction Regulation & Knowledge Distillation

2024-09-29 · Huidong Tang, Chen Li, Huachong Yu, Sayaka Kamei, Yasuhiko Morimoto

Federated learning (FL) has emerged as a transformative training paradigm, particularly invaluable in privacy-sensitive domains like healthcare. However, client heterogeneity in data, computing power, and tasks poses a significant challenge. To address such a challenge, we propose an FL optimization algorithm that integrates model delta regularization, personalized models, federated knowledge distillation, and mix-pooling. Model delta regularization optimizes model updates centrally on the server, efficiently updating clients with minimal communication costs. Personalized models and federated knowledge distillation strategies are employed to tackle task heterogeneity effectively. Additionally, mix-pooling is introduced to accommodate variations in the sensitivity of readout operations. Experimental results demonstrate the remarkable accuracy and rapid convergence achieved by model delta regularization. Additionally, the federated knowledge distillation algorithm notably improves FL performance, especially in scenarios with diverse data. Moreover, mix-pooling readout operations provide tangible benefits for clients, showing the effectiveness of our proposed methods.

📄 PDF Abstract BibTeX arXiv:2409.19741

Code (0)

등록된 구현이 없습니다.

Tasks

Federated Learning Knowledge Distillation

Methods 이 논문이 사용한 방법론

Knowledge Distillation A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions.…

FedMCP: Parameter-Efficient Federated Learning with Model-Contrastive Personalization

2024-08-28 · Qianyi Zhao, Chen Qu, Cen Chen, Mingyuan Fan 외

With increasing concerns and regulations on data privacy, fine-tuning pretrained language models (PLMs) in federated learning (FL) has become a common paradigm for NLP tasks. Despite being extensively studied, the existi…

Federated Learning parameter-efficient fine-tuning

BiFedKD: Bidirectional Federated Knowledge Distillation Framework for Non-IID and Long-Tailed ECG Monitoring

2026-05-14 · Zixuan Shu, Tiancheng Cao, Hen-Wei Huang arxiv

Electrocardiogram (ECG) monitoring in Internet of Medical Things (IoMT) networks is constrained by strict data-sharing regulations and privacy concerns. Federated learning (FL) enables collaborative learning by keeping r…

Knowledge Distillation Federated Learning

Bringing Multi-Modal Multi-Task Federated Foundation Models to Education Domain: Prospects and Challenges

2025-09-09 · Kasra Borazjani, Naji Khosravan, Rajeev Sahay, Bita Akram 외 arxiv

Multi-modal multi-task (M3T) foundation models (FMs) have recently shown transformative potential in artificial intelligence, with emerging applications in education. However, their deployment in real-world educational s…

Federated Learning Continual Learning

From Distributed Machine Learning to Federated Learning: A Survey

2021-04-29 · Ji Liu, Jizhou Huang, Yang Zhou, Xuhong LI 외

In recent years, data and computing resources are typically distributed in the devices of end users, various regions or organizations. Because of laws or regulations, the distributed data and computing resources cannot b…

BIG-bench Machine Learning Federated Learning Survey

Federated Knowledge Graph Unlearning via Diffusion Model

2024-03-13 · Bingchen Liu, Yuanyuan Fang

Federated learning (FL) promotes the development and application of artificial intelligence technologies by enabling model sharing and collaboration while safeguarding data privacy. Knowledge graph (KG) embedding represe…

Federated Learning Knowledge Graphs Machine Unlearning model