Practical Locally Private Federated Learning with Communication Efficiency
Federated learning (FL) is a technique that trains machine learning models from decentralized data sources. We study FL under local differential privacy constraints, which provides strong protection against sensitive data disclosures via obfuscating the data before leaving the client. We identify two major concerns in designing practical privacy-preserving FL algorithms: communication efficiency and high-dimensional compatibility. We then develop a gradient-based learning algorithm called \emph{sqSGD} (selective quantized stochastic gradient descent) that addresses both concerns. The proposed algorithm is based on a novel privacy-preserving quantization scheme that uses a constant number of bits per dimension per client. Then we improve the base algorithm in two ways: first, we apply a gradient subsampling strategy that offers simultaneously better training performance and smaller communication costs under a fixed privacy budget. Secondly, we utilize randomized rotation as a preprocessing step to reduce quantization error. We also initialize a discussion about the role of quantization and perturbation in FL algorithm design with privacy and communication constraints. Finally, the practicality of the proposed framework is demonstrated on benchmark datasets. Experiment results show that sqSGD successfully learns large models like LeNet and ResNet with local privacy constraints. In addition, with fixed privacy and communication level, the performance of sqSGD significantly dominates baselines that do not involve quantization.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningPrivacy PreservingQuantizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models
Federated Learning (FL) has garnered increasing attention due to its unique characteristic of allowing heterogeneous clients to process their private data locally and interact with a central server, while being respectfu…
Federated LearningQuantizationFedSpaLLM: Federated Pruning of Large Language Models
Large Language Models (LLMs) achieve state-of-the-art performance but are challenging to deploy due to their high computational and storage demands. Pruning can reduce model size, yet existing methods assume public acces…
Federated LearningPrivacy PreservingPractical and Light-weight Secure Aggregation for Federated Submodel Learning
Recently, Niu, et. al. introduced a new variant of Federated Learning (FL), called Federated Submodel Learning (FSL). Different from traditional FL, each client locally trains the submodel (e.g., retrieved from the serve…
Federated LearningPrivacy PreservingRetrievalsqSGD: Locally Private and Communication Efficient Federated Learning
Federated learning (FL) is a technique that trains machine learning models from decentralized data sources. We study FL under local notions of privacy constraints, which provides strong protection against sensitive data …
Federated LearningPrivacy PreservingQuantizationLLM-QFL: Distilling Large Language Model for Quantum Federated Learning
Inspired by the power of large language models (LLMs), our research adapts them to quantum federated learning (QFL) to boost efficiency and performance. We propose a federated fine-tuning method that distills an LLM with…
Federated LearningLanguage ModelingLanguage ModellingLarge Language Model