FedTLU: Federated Learning with Targeted Layer Updates
Federated learning (FL) addresses privacy concerns in training language models by enabling multiple clients to contribute to the training, without sending their data to others. However, non-IID (identically and independently distributed) data across clients often limits FL's performance. This issue is especially challenging during model fine-tuning, as noise due to variations in clients' data distributions can harm model convergence near stationary points. This paper proposes a targeted layer update strategy for fine-tuning in FL. Instead of randomly updating layers of the language model, as often done in practice, we use a scoring mechanism to identify and update the most critical layers, avoiding excessively noisy or even poisoned updates by freezing the parameters in other layers. We show in extensive experiments that our method improves convergence and performance in non-IID settings, offering a more efficient approach to fine-tuning federated language models.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Learning to Detect Malicious Clients for Robust Federated Learning
Federated learning systems are vulnerable to attacks from malicious clients. As the central server in the system cannot govern the behaviors of the clients, a rogue client may initiate an attack by sending malicious mode…
Federated Learningimage-classificationImage ClassificationModel Poisoning+1Dynamic Meta-Layer Aggregation for Byzantine-Robust Federated Learning
Federated Learning (FL) is increasingly applied in sectors like healthcare, finance, and IoT, enabling collaborative model training while safeguarding user privacy. However, FL systems are susceptible to Byzantine advers…
Computational EfficiencyFederated LearningCeltibero: Robust Layered Aggregation for Federated Learning
Federated Learning (FL) is an innovative approach to distributed machine learning. While FL offers significant privacy advantages, it also faces security challenges, particularly from poisoning attacks where adversaries …
Federated LearningUA-FedRec: Untargeted Attack on Federated News Recommendation
News recommendation is critical for personalized news distribution. Federated news recommendation enables collaborative model learning from many clients without sharing their raw data. It is promising for privacy-preserv…
Federated LearningNews RecommendationPrivacy PreservingRecommendation SystemsPoisoning Federated Recommender Systems with Fake Users
Federated recommendation is a prominent use case within federated learning, yet it remains susceptible to various attacks, from user to server-side vulnerabilities. Poisoning attacks are particularly notable among user-s…
Federated LearningRecommendation Systems