GTAdam: Gradient Tracking with Adaptive Momentum for Distributed Online Optimization
This paper deals with a network of computing agents aiming to solve an online optimization problem in a distributed fashion, i.e., by means of local computation and communication, without any central coordinator. We propose the gradient tracking with adaptive momentum estimation (GTAdam) distributed algorithm, which combines a gradient tracking mechanism with first and second order momentum estimates of the gradient. The algorithm is analyzed in the online setting for strongly convex cost functions with Lipschitz continuous gradients. We provide an upper bound for the dynamic regret given by a term related to the initial conditions and another term related to the temporal variations of the objective functions. Moreover, a linear convergence rate is guaranteed in the static setup. The algorithm is tested on a time-varying classification problem, on a (moving) target localization problem, and in a stochastic optimization setup from image classification. In these numerical experiments from multi-agent learning, GTAdam outperforms state-of-the-art distributed optimization methods.
Code (0)
등록된 구현이 없습니다.
Tasks
Distributed OptimizationGeneral Classificationimage-classificationImage ClassificationStochastic OptimizationSimilar Papers 제목 키워드 기반
Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Data
SGD with momentum is one of the key components for improving the performance of neural networks. For decentralized learning, a straightforward approach using momentum is Distributed SGD (DSGD) with momentum (DSGDm). Howe…
Image ClassificationOn the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization
In this paper, we provide a comprehensive theoretical analysis of Stochastic Gradient Descent (SGD) and its momentum variants (Polyak Heavy-Ball and Nesterov) for tracking time-varying optima under strong convexity and s…
Stochastic OptimizationOrdered Momentum for Asynchronous SGD
Distributed learning is essential for training large-scale deep models. Asynchronous SGD (ASGD) and its variants are commonly used distributed learning methods, particularly in scenarios where the computing capabilities …
Momentum Accelerates the Convergence of Stochastic AUPRC Maximization
In this paper, we study stochastic optimization of areas under precision-recall curves (AUPRC), which is widely used for combating imbalanced classification tasks. Although a few methods have been proposed for maximizing…
imbalanced classificationStochastic OptimizationCompressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimization
In this paper, we design two compressed decentralized algorithms for solving nonconvex stochastic optimization under two different scenarios. Both algorithms adopt a momentum technique to achieve fast convergence and a m…
Stochastic Optimization