Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization
Distributed and federated learning are essential paradigms for training models across decentralized data sources while preserving privacy, yet communication overhead remains a major bottleneck. This dissertation explores strategies to improve communication efficiency, focusing on model compression, local training, and personalization. We establish a unified framework for biased and unbiased compression operators with convergence guarantees, then propose adaptive local training strategies that incorporate personalization to accelerate convergence and mitigate client drift. In particular, Scafflix balances global and personalized objectives, achieving superior performance under both IID and non-IID settings. We further introduce privacy-preserving pruning frameworks that optimize sparsity while minimizing communication costs, with Cohort-Squeeze leveraging hierarchical aggregation to reduce cross-device overhead. Finally, SymWanda, a symmetric post-training pruning method, enhances robustness under high sparsity and maintains accuracy without retraining. Extensive experiments on benchmarks and large-scale language models demonstrate favorable trade-offs among accuracy, convergence, and communication, offering theoretical and practical insights for scalable, efficient distributed learning.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningModel CompressionSimilar Papers 제목 키워드 기반
Fed-DLoRA: Efficient Wireless Federated Learning with Dynamic Low-Rank Adaptation
Federated learning (FL) offers a promising distributed learning paradigm for internet of vehicles (IoV) applications. However, it faces challenges from communication overhead and dynamic environments. Model compression t…
Federated LearningModel CompressionProvably Doubly Accelerated Federated Learning: The First Theoretically Successful Combination of Local Training and Communication Compression
In federated learning, a large number of users are involved in a global learning task, in a collaborative way. They alternate local computations and two-way communication with a distant orchestrating server. Communicatio…
Distributed OptimizationFederated LearningBiCoLoR: Communication-Efficient Optimization with Bidirectional Compression and Local Training
Slow and costly communication is often the main bottleneck in distributed optimization, especially in federated learning where it occurs over wireless networks. We introduce BiCoLoR, a communication-efficient optimizatio…
Distributed OptimizationFederated LearningBandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
Current data compression methods, such as sparsification in Federated Averaging (FedAvg), effectively enhance the communication efficiency of Federated Learning (FL). However, these methods encounter challenges such as t…
Data CompressionFederated LearningFairEnergy: Contribution-Based Fairness meets Energy Efficiency in Federated Learning
Federated learning (FL) enables collaborative model training across distributed devices while preserving data privacy. However, balancing energy efficiency and fair participation while ensuring high model accuracy remain…
Federated Learning