Adaptive Sampling and Clipping for Private Worst-Case Group Optimization
A central requirement for the acceptance of machine learning methods for human-centric tasks is that they should be fair, in the sense that they should work comparably well for individuals from different societal groups. A second, equally important, requirement is that they should respect the privacy of user data. While techniques exist to address each aspect in isolation, such as worst-case group optimization for the former and differentially private SGD for the latter, these are often at odds with with each other, and no practical method currently exists to enforce both requirements simultaneously. In this work, we overcome this problem and propose an algorithm for optimizing the worst-case group accuracy in a differentially private way. Our main contribution is ASC (Adaptively Sampled and Clipped Worst-case Group Optimization), which adaptively controls both the sampling rate and the clipping threshold of each group's gradient contributions. Thereby, it is able to reweight the training objective in favor of harder-to-learn groups, while keeping the noise required to enforce privacy low enough to preserve model utility. Our experiments show that ASC achieves substantially higher worst-case group accuracy than prior work, without sacrificing overall average accuracy.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Mitigating Disparate Impact of Differentially Private Learning through Bounded Adaptive Clipping
Differential privacy (DP) has become an essential framework for privacy-preserving machine learning. Existing DP learning methods, however, often have disparate impacts on model predictions, e.g., for minority groups. Gr…
Privacy PreservingPrivate Adaptive Gradient Methods for Convex Optimization
We study adaptive methods for differentially private convex optimization, proposing and analyzing differentially private variants of a Stochastic Gradient Descent (SGD) algorithm with adaptive stepsizes, as well as the A…
Improved Worst-Case Regret Bounds for Randomized Least-Squares Value Iteration
This paper studies regret minimization with randomized value functions in reinforcement learning. In tabular finite-horizon Markov Decision Processes, we introduce a clipping variant of one classical Thompson Sampling (T…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)Thompson SamplingDP-MacAdam: Differentially Private Mechanism with Adaptive Clipping and Adaptive Momentum
Differentially private stochastic gradient descent (DP-SGD) has become the standard framework for privacy-preserving machine learning, yet its reliance on a fixed gradient clipping threshold to limit sensitivity remains …
Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noise
We develop a worst-case complexity theory for stochastically preconditioned stochastic gradient descent (SPSGD) and its accelerated variants under heavy-tailed noise, a setting that encompasses widely used adaptive metho…