Efficient Language Model Architectures for Differentially Private Federated Learning

2024-03-12 · Jae Hun Ro, Srinadh Bhojanapalli, Zheng Xu, Yanxiang Zhang, Ananda Theertha Suresh

Cross-device federated learning (FL) is a technique that trains a model on data distributed across typically millions of edge devices without data leaving the devices. SGD is the standard client optimizer for on device training in cross-device FL, favored for its memory and computational efficiency. However, in centralized training of neural language models, adaptive optimizers are preferred as they offer improved stability and performance. In light of this, we ask if language models can be modified such that they can be efficiently trained with SGD client optimizers and answer this affirmatively. We propose a scale-invariant Coupled Input Forget Gate (SI CIFG) recurrent network by modifying the sigmoid and tanh activations in the recurrent cell and show that this new model converges faster and achieves better utility than the standard CIFG recurrent model in cross-device FL in large scale experiments. We further show that the proposed scale invariant modification also helps in federated learning of larger transformer models. Finally, we demonstrate the scale invariant modification is also compatible with other non-adaptive algorithms. Particularly, our results suggest an improved privacy utility trade-off in federated learning with differential privacy.

📄 PDF Abstract BibTeX arXiv:2403.08100

Code (0)

등록된 구현이 없습니다.

Tasks

Computational Efficiency Federated LearningLanguage ModelingLanguage Modelling

Methods 이 논문이 사용한 방법론

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Can Public Large Language Models Help Private Cross-device Federated Learning?

2023-05-20 · Boxin Wang, Yibo Jacky Zhang, Yuan Cao, Bo Li 외

We study (differentially) private federated learning (FL) of language models. The language models in cross-device FL are relatively small, which can be trained with meaningful formal user-level differential privacy (DP) …

Federated Learning

Defending against Reconstruction Attacks through Differentially Private Federated Learning for Classification of Heterogeneous Chest X-Ray Data

2022-05-06 · Joceline Ziegler, Bjarne Pfitzner, Heinrich Schulz, Axel Saalbach 외

Privacy regulations and the physical distribution of heterogeneous data are often primary concerns for the development of deep learning models in a medical context. This paper evaluates the feasibility of differentially …

Binary Classification Federated Learning Image Reconstruction X-ray Classification

LanFL: Differentially Private Federated Learning with Large Language Models using Synthetic Samples

2024-10-24 · Huiyu Wu, Diego Klabjan

Federated Learning (FL) is a collaborative, privacy-preserving machine learning framework that enables multiple participants to train a single global model. However, the recent advent of powerful Large Language Models (L…

Federated Learning Privacy Preserving

Efficient Differentially Private Secure Aggregation for Federated Learning via Hardness of Learning with Errors

2021-12-13 · Timothy Stevens, Christian Skalka, Christelle Vincent, John Ring 외

Federated machine learning leverages edge computing to develop models from network user data, but privacy in federated learning remains a major challenge. Techniques using differential privacy have been proposed to addre…

Edge-computing Federated Learning

Learning Rate Adaptation for Federated and Differentially Private Learning

2018-09-11 · Antti Koskela, Antti Honkela

We propose an algorithm for the adaptation of the learning rate for stochastic gradient descent (SGD) that avoids the need for validation set use. The idea for the adaptiveness comes from the technique of extrapolation: …