LanFL: Differentially Private Federated Learning with Large Language Models using Synthetic Samples
Federated Learning (FL) is a collaborative, privacy-preserving machine learning framework that enables multiple participants to train a single global model. However, the recent advent of powerful Large Language Models (LLMs) with tens to hundreds of billions of parameters makes the naive application of traditional FL methods to LLMs impractical due to high computational and communication costs. Furthermore, end users of LLMs often lack access to full architectures and weights of the models, making it impossible for participants to fine-tune these models directly. This paper introduces a novel FL scheme for LLMs, named LanFL, which is purely prompt-based and treats the underlying LLMs as black boxes. We have developed a differentially private synthetic sample generation mechanism to facilitate knowledge sharing among participants, along with a prompt optimization scheme that enables learning from synthetic samples. Our extensive experiments demonstrate that LanFL successfully facilitates learning among participants while preserving the privacy of local datasets across various tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
Federated LearningPrivacy PreservingSimilar Papers 제목 키워드 기반
Hierarchical Federated Learning through LAN-WAN Orchestration
Federated learning (FL) was designed to enable mobile phones to collaboratively learn a global model without uploading their private data to a cloud server. However, exiting FL protocols has a critical communication bott…
Federated LearningCan Public Large Language Models Help Private Cross-device Federated Learning?
We study (differentially) private federated learning (FL) of language models. The language models in cross-device FL are relatively small, which can be trained with meaningful formal user-level differential privacy (DP) …
Federated LearningEfficient Differentially Private Secure Aggregation for Federated Learning via Hardness of Learning with Errors
Federated machine learning leverages edge computing to develop models from network user data, but privacy in federated learning remains a major challenge. Techniques using differential privacy have been proposed to addre…
Edge-computingFederated LearningPrivately Customizing Prefinetuning to Better Match User Data in Federated Learning
In Federated Learning (FL), accessing private client data incurs communication and privacy costs. As a result, FL deployments commonly prefinetune pretrained foundation models on a (large, possibly public) dataset that i…
Federated LearningLanguage ModelingLanguage ModellingLarge Language Model+1Federated Learning in Non-IID Settings Aided by Differentially Private Synthetic Data
Federated learning (FL) is a privacy-promoting framework that enables potentially large number of clients to collaboratively train machine learning models. In a FL system, a server coordinates the collaboration by collec…
Federated Learningimage-classificationImage Classification