Differentially Private Gradient-Tracking-Based Distributed Stochastic Optimization over Directed Graphs
This paper proposes a new differentially private gradient-tracking-based distributed stochastic optimization algorithm over directed graphs. Specifically, privacy noises are added to each agent's state and tracking variable to prevent information leakage, and then perturbed states and tracking variables are transmitted to neighbors. We design two novel schemes of the iteration step-sizes and the sampling number for the algorithm. By using the sampling parameter-controlled subsampling method, both schemes enhance the differential privacy level, and achieve the finite cumulative privacy budget even over infinite iterations. The convergence rate of the algorithm is shown for both nonconvex with the Polyak-Lojasiewicz condition and strongly convex objectives: Scheme (S1) achieves the polynomial convergence rate, and Scheme (S2) achieves the exponential convergence rate. The trade-off between the privacy and the convergence rate is presented. The algorithm's effectiveness and superior performance over the existing works are demonstrated through numerical examples of distributed training on benchmark datasets "MNIST" and "CIFAR-10".
Code (0)
등록된 구현이 없습니다.
Tasks
Stochastic OptimizationSimilar Papers 제목 키워드 기반
Differentially Private Distributed Stochastic Optimization with Time-Varying Sample Sizes
Differentially private distributed stochastic optimization has become a hot topic due to the urgent need of privacy protection in distributed stochastic optimization. In this paper, two-time scale stochastic approximatio…
Stochastic OptimizationDifferentially Private Dual Gradient Tracking for Distributed Resource Allocation
This paper investigates privacy issues in distributed resource allocation over directed networks, where each agent holds a private cost function and optimizes its decision subject to a global coupling constraint through …
Dopamine: Differentially Private Federated Learning on Medical Data
While rich medical datasets are hosted in hospitals distributed across the world, concerns on patients' privacy is a barrier against using such data to train deep neural networks (DNNs) for medical diagnostics. We propos…
Federated LearningAdaptive Differentially Private Empirical Risk Minimization
We propose an adaptive (stochastic) gradient perturbation method for differentially private empirical risk minimization. At each iteration, the random noise added to the gradient is optimally adapted to the stepsize; we …
Second-Order Convergence in Private Stochastic Non-Convex Optimization
We investigate the problem of finding second-order stationary points (SOSP) in differentially private (DP) stochastic non-convex optimization. Existing methods suffer from two key limitations: (i) inaccurate convergence …
Model Selection