Backpropagation Clipping for Deep Learning with Differential Privacy
We present backpropagation clipping, a novel variant of differentially private stochastic gradient descent (DP-SGD) for privacy-preserving deep learning. Our approach clips each trainable layer's inputs (during the forward pass) and its upstream gradients (during the backward pass) to ensure bounded global sensitivity for the layer's gradient; this combination replaces the gradient clipping step in existing DP-SGD variants. Our approach is simple to implement in existing deep learning frameworks. The results of our empirical evaluation demonstrate that backpropagation clipping provides higher accuracy at lower values for the privacy parameter $\epsilon$ compared to previous work. We achieve 98.7% accuracy for MNIST with $\epsilon = 0.07$ and 74% accuracy for CIFAR-10 with $\epsilon = 3.64$.
Code (1)
Tasks
Deep LearningPrivacy PreservingPrivacy Preserving Deep LearningSensitivityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
An Adaptive Differentially Private Federated Learning Framework
Federated learning enables collaborative model training across distributed clients while preserving data privacy. However, in practical deployments, device heterogeneity and non-independent and identically distributed (N…
Dimensionality ReductionFederated LearningExploring the Limits of Differentially Private Deep Learning with Group-wise Clipping
Differentially private deep learning has recently witnessed advances in computational efficiency and privacy-utility trade-off. We explore whether further improvements along the two axes are possible and provide affirmat…
Computational EfficiencyRemoving Disparate Impact of Differentially Private Stochastic Gradient Descent on Model Accuracy
When we enforce differential privacy in machine learning, the utility-privacy trade-off is different w.r.t. each group. Gradient clipping and random noise addition disproportionately affect underrepresented and complex c…
Multi-Objective Optimization for Privacy-Utility Balance in Differentially Private Federated Learning
Federated learning (FL) enables collaborative model training across distributed clients without sharing raw data, making it a promising approach for privacy-preserving machine learning. However, ensuring differential pri…
Federated LearningPrivacy PreservingUnderstanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacy
Providing privacy protection has been one of the primary motivations of Federated Learning (FL). Recently, there has been a line of work on incorporating the formal privacy notion of differential privacy with FL. To guar…
Federated Learning