VaultGemma: A Differentially Private Gemma Model
We introduce VaultGemma 1B, a 1 billion parameter model within the Gemma family, fully trained with differential privacy. Pretrained on the identical data mixture used for the Gemma 2 series, VaultGemma 1B represents a significant step forward in privacy-preserving large language models. We openly release this model to the community
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling
Preference-based fine-tuning has become an important component in training large language models, and the data used at this stage may contain sensitive user information. A central question is how to design a differential…
Reinforcement LearningDifferentially Private In-context Learning via Sampling Few-shot Mixed with Zero-shot Outputs
In-context learning (ICL) has shown promising improvement in downstream task adaptation of LLMs by augmenting prompts with relevant input-output examples (demonstrations). However, the ICL demonstrations can contain priv…
Few-Shot LearningIn-Context LearningReconstruction of Differentially Private Text Sanitization via Large Language Models
Differential privacy (DP) is the de facto privacy standard against privacy leakage attacks, including many recently discovered ones against large language models (LLMs). However, we discovered that LLMs could reconstruct…
Differentially Private Algorithms for Empirical Machine Learning
An important use of private data is to build machine learning classifiers. While there is a burgeoning literature on differentially private classification algorithms, we find that they are not practical in real applicati…
BIG-bench Machine LearningGeneral ClassificationAdaptive Differentially Private Empirical Risk Minimization
We propose an adaptive (stochastic) gradient perturbation method for differentially private empirical risk minimization. At each iteration, the random noise added to the gradient is optimally adapted to the stepsize; we …