DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)
The Adam optimizer is a popular choice in contemporary deep learning, due to its strong empirical performance. However we observe that in privacy sensitive scenarios, the traditional use of Differential Privacy (DP) with the Adam optimizer leads to sub-optimal performance on several tasks. We find that this performance degradation is due to a DP bias in Adam's second moment estimator, introduced by the addition of independent noise in the gradient computation to enforce DP guarantees. This DP bias leads to a different scaling for low variance parameter updates, that is inconsistent with the behavior of non-private Adam. We propose DP-AdamBC, an optimization algorithm which removes the bias in the second moment estimation and retrieves the expected behaviour of Adam. Empirically, DP-AdamBC significantly improves the optimization performance of DP-Adam by up to 3.5% in final accuracy in image, text, and graph node classification tasks.
Code (1)
Tasks
Node ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
DP-AdamW: Investigating Decoupled Weight Decay and Bias Correction in Private Deep Learning
As deep learning methods increasingly utilize sensitive data on a widespread scale, differential privacy (DP) offers formal guarantees to protect against information leakage during model training. A significant challenge…
Image ClassificationNode ClassificationText ClassificationIs your batch size the problem? Revisiting the Adam-SGD gap in language modeling
Adam is known to perform significantly better than Stochastic Gradient Descent (SGD) in language models, a phenomenon for which a number of explanations have been proposed. In this work, we revisit this "optimizer gap" t…
Language ModelingLanguage ModellingOn the Performance of Differentially Private Optimization with Heavy-Tail Class Imbalance
In this work, we analyze the optimization behaviour of common private learning optimization algorithms under heavy-tail class imbalanced distribution. We show that, in a stylized model, optimizing with Gradient Descent w…
ADOPT: Modified Adam Can Converge with Any $β_2$ with the Optimal Rate
Adam is one of the most popular optimization algorithms in deep learning. However, it is known that Adam does not converge in theory unless choosing a hyperparameter, i.e., $\beta_2$, in a problem-dependent manner. There…
Deep Reinforcement Learningimage-classificationImage ClassificationDoes Adam Converge and When?
In this blog post, we revisit the (non-)convergence behavior of Adam. Especially, we briefly review the non-convergence results by Reddi et al'19 and the convergence results by Shi et al.'20. Their results take important…