AdaCliP: Adaptive Clipping for Private SGD
Privacy preserving machine learning algorithms are crucial for learning models over user data to protect sensitive information. Motivated by this, differentially private stochastic gradient descent (SGD) algorithms for training machine learning models have been proposed. At each step, these algorithms modify the gradients and add noise proportional to the sensitivity of the modified gradients. Under this framework, we propose AdaCliP, a theoretically motivated differentially private SGD algorithm that provably adds less noise compared to the previous methods, by using coordinate-wise adaptive clipping of the gradient. We empirically demonstrate that AdaCliP reduces the amount of added noise and produces models with better accuracy.
Code (1)
Tasks
BIG-bench Machine LearningPrivacy PreservingSensitivityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SoftAdaClip: A Smooth Clipping Strategy for Fair and Private Model Training
Differential privacy (DP) provides strong protection for sensitive data, but often reduces model performance and fairness, especially for underrepresented groups. One major reason is gradient clipping in DP-SGD, which ca…
DP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum
Differentially private stochastic gradient descent (DP-SGD) has become the standard framework for privacy-preserving machine learning, yet its reliance on a fixed gradient clipping threshold to limit sensitivity remains …
Exploring the Limits of Differentially Private Deep Learning with Group-wise Clipping
Differentially private deep learning has recently witnessed advances in computational efficiency and privacy-utility trade-off. We explore whether further improvements along the two axes are possible and provide affirmat…
Computational EfficiencyDP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models
Federated learning (FL) enables the collaborative training of large-scale language models (LLMs) across edge devices while keeping user data on-device. However, FL still exposes sensitive information through client-provi…
Federated LearningOn the Convergence of DP-SGD with Adaptive Clipping
Stochastic Gradient Descent (SGD) with gradient clipping is a powerful technique for enabling differentially private optimization. Although prior works extensively investigated clipping with a constant threshold, private…