Privately Aligning Language Models with Reinforcement Learning
Positioned between pre-training and user deployment, aligning large language models (LLMs) through reinforcement learning (RL) has emerged as a prevailing strategy for training instruction following-models such as ChatGPT. In this work, we initiate the study of privacy-preserving alignment of LLMs through Differential Privacy (DP) in conjunction with RL. Following the influential work of Ziegler et al. (2020), we study two dominant paradigms: (i) alignment via RL without human in the loop (e.g., positive review generation) and (ii) alignment via RL from human feedback (RLHF) (e.g., summarization in a human-preferred way). We give a new DP framework to achieve alignment via RL, and prove its correctness. Our experimental results validate the effectiveness of our approach, offering competitive utility while ensuring strong privacy protections.
Code (0)
등록된 구현이 없습니다.
Tasks
Instruction FollowingPrivacy Preservingreinforcement-learningReinforcement LearningReinforcement Learning (RL)Review GenerationSimilar Papers 제목 키워드 기반
Privacy-Preserving Instructions for Aligning Large Language Models
Service providers of large language model (LLM) applications collect user instructions in the wild and use them in further aligning LLMs with users' intentions. These instructions, which potentially contain sensitive inf…
Language ModelingLanguage ModellingLarge Language ModelPrivacy PreservingPrivately generating tabular data using language models
Privately generating synthetic data from a table is an important brick of a privacy-first world. We propose and investigate a simple approach of treating each row in a table as a sentence and training a language model wi…
Language ModelingLanguage ModellingSentenceA Contextual-Bandit Oversight Game with Two-Sided Informational Asymmetry
We study runtime human oversight of an AI agent when private information runs in both directions: the human privately knows her reward function, while the AI privately knows the quality of the action it proposes. This is…
Reinforcement LearningPrivately Learning Mixtures of Axis-Aligned Gaussians
We consider the problem of learning mixtures of Gaussians under the constraint of approximate differential privacy. We prove that $\widetilde{O}(k^2 d \log^{3/2}(1/\delta) / \alpha^2 \varepsilon)$ samples are sufficient …
Privately Customizing Prefinetuning to Better Match User Data in Federated Learning
In Federated Learning (FL), accessing private client data incurs communication and privacy costs. As a result, FL deployments commonly prefinetune pretrained foundation models on a (large, possibly public) dataset that i…
Federated LearningLanguage ModelingLanguage ModellingLarge Language Model+1