Differential Privacy Guarantees for Stochastic Gradient Langevin Dynamics
We analyse the privacy leakage of noisy stochastic gradient descent by modeling R\'enyi divergence dynamics with Langevin diffusions. Inspired by recent work on non-stochastic algorithms, we derive similar desirable properties in the stochastic setting. In particular, we prove that the privacy loss converges exponentially fast for smooth and strongly convex objectives under constant step size, which is a significant improvement over previous DP-SGD analyses. We also extend our analysis to arbitrary sequences of varying step sizes and derive new utility bounds. Last, we propose an implementation and our experiments show the practical utility of our approach compared to classical DP-SGD libraries.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Can Stochastic Gradient Langevin Dynamics Provide Differential Privacy for Deep Learning?
Bayesian learning via Stochastic Gradient Langevin Dynamics (SGLD) has been suggested for differentially private learning. While previous research provides differential privacy bounds for SGLD at the initial steps of the…
Differential privacy guarantees of Markov chain Monte Carlo algorithms
This paper aims to provide differential privacy (DP) guarantees for Markov chain Monte Carlo (MCMC) algorithms. In a first part, we establish DP guarantees on samples output by MCMC algorithms as well as Monte Carlo esti…
Privacy PreservingLangevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning
Machine unlearning has raised significant interest with the adoption of laws ensuring the ``right to be forgotten''. Researchers have provided a probabilistic notion of approximate unlearning under a similar definition o…
Machine UnlearningDimension Independent Generalization of DP-SGD for Overparameterized Smooth Convex Optimization
This paper considers the generalization performance of differentially private convex learning. We demonstrate that the convergence analysis of Langevin algorithms can be used to obtain new generalization bounds with diff…
Generalization BoundsA Better Bound Gives a Hundred Rounds: Enhanced Privacy Guarantees via $f$-Divergences
We derive the optimal differential privacy (DP) parameters of a mechanism that satisfies a given level of R\'enyi differential privacy (RDP). Our result is based on the joint range of two $f$-divergences that underlie th…