Papers Distributed Optimization
“Distributed Optimization” 태그가 달린 논문 601편 · 필터 해제
99% of Distributed Optimization is a Waste of Time: The Issue and How to Fix it
Many popular distributed optimization methods for training machine learning models fit the following template: a local gradient estimate is computed independently by each worker, then communicated to a master, which subs…
Distributed OptimizationTrajectory Normalized Gradients for Distributed Optimization
Recently, researchers proposed various low-precision gradient compression, for efficient communication in large-scale distributed optimization. Based on these work, we try to reduce the communication complexity from a ne…
BenchmarkingDistributed OptimizationDistributed Nesterov gradient methods over arbitrary graphs
In this letter, we introduce a distributed Nesterov method, termed as $\mathcal{ABN}$, that does not require doubly-stochastic weight matrices. Instead, the implementation is based on a simultaneous application of both r…
Distributed OptimizationA continuous-time analysis of distributed stochastic gradient
We analyze the effect of synchronization on distributed stochastic gradient algorithms. By exploiting an analogy with dynamical models of biological quorum sensing - where synchronization between agents is induced throug…
Distributed OptimizationHyperspectral Unmixing Based on Clustered Multitask Networks
Hyperspectral remote sensing is a prominent research topic in data processing. Most of the spectral unmixing algorithms are developed by adopting the linear mixing models. Nonnegative matrix factorization (NMF) and its d…
Distributed OptimizationHyperspectral UnmixingStochastic Distributed Optimization for Machine Learning from Decentralized Features
Distributed machine learning has been widely studied in the literature to scale up machine learning model training in the presence of an ever-increasing amount of data. We study distributed machine learning from another …
BIG-bench Machine LearningDistributed OptimizationFederated Optimization in Heterogeneous Networks
Federated Learning is a distributed learning paradigm with two key challenges that differentiate it from traditional distributed optimization: (1) significant variability in terms of the systems characteristics on each d…
Distributed OptimizationFederated LearningSolving Non-smooth Constrained Programs with Lower Complexity than \mathcal{O}(1/\varepsilon): A Primal-Dual Homotopy Smoothing Approach
We propose a new primal-dual homotopy smoothing algorithm for a linearly constrained convex program, where neither the primal nor the dual function has to be smooth or strongly convex. The best known iteration complexity…
Distributed OptimizationMarkov Chain Block Coordinate Descent
The method of block coordinate gradient descent (BCD) has been a powerful method for large-scale optimization. This paper considers the BCD method that successively updates a series of blocks selected according to a Mark…
Distributed OptimizationDistributed Convex Optimization With Limited Communications
In this paper, a distributed convex optimization algorithm, termed \emph{distributed coordinate dual averaging} (DCDA) algorithm, is proposed. The DCDA algorithm addresses the scenario of a large distributed optimization…
Distributed OptimizationvalidDistributed optimization in wireless sensor networks: an island-model framework
Wireless Sensor Networks (WSNs) is an emerging technology in several application domains, ranging from urban surveillance to environmental and structural monitoring. Computational Intelligence (CI) techniques are particu…
Anomaly DetectionCPUDistributed OptimizationSparsified SGD with Memory
Huge scale machine learning problems are nowadays tackled by distributed optimization algorithms, i.e. algorithms that leverage the compute power of many devices for training. The communication overhead is a key bottlene…
Distributed OptimizationQuantizationA Dual Approach for Optimal Algorithms in Distributed Optimization over Networks
We study dual-based algorithms for distributed convex optimization problems over networks, where the objective is to minimize a sum $\sum_{i=1}^{m}f_i(z)$ of functions over in a network. We provide complexity bounds for …
Distributed OptimizationGradient Primal-Dual Algorithm Converges to Second-Order Stationary Solution for Nonconvex Distributed Optimization Over Networks
In this work, we study two first-order primal-dual based algorithms, the Gradient Primal-Dual Algorithm (GPDA) and the Gradient Alternating Direction Method of Multipliers (GADMM), for solving a class of linearly co…
Distributed OptimizationAn Exact Quantized Decentralized Gradient Descent Algorithm
We consider the problem of decentralized consensus optimization, where the sum of $n$ smooth and strongly convex functions are minimized over $n$ distributed agents that form a connected network. In particular, we consid…
Distributed OptimizationQuantizationA Tight Convergence Analysis for Stochastic Gradient Descent with Delayed Updates
We provide tight finite-time convergence bounds for gradient descent and stochastic gradient descent on quadratic functions, when the gradients are delayed and reflect iterates from $\tau$ rounds ago. First, we show that…
Distributed OptimizationError Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization
Large-scale distributed optimization is of great importance in various applications. For data-parallel based distributed learning, the inter-node gradient communication often becomes the performance bottleneck. In this p…
Distributed OptimizationQuantizationA Distributed Second-Order Algorithm You Can Trust
Due to the rapid growth of data and computational resources, distributed optimization has become an active research area in recent years. While first-order methods seem to dominate the field, second-order methods are nev…
Distributed OptimizationSecond-order methodsDistributed learning with compressed gradients
Asynchronous computation and gradient compression have emerged as two key techniques for achieving scalability in distributed optimization for large-scale machine learning. This paper presents a unified analysis framewor…
BIG-bench Machine LearningDistributed OptimizationDouble Quantization for Communication-Efficient Distributed Optimization
Modern distributed training of machine learning models suffers from high communication overhead for synchronizing stochastic gradients and model parameters. In this paper, to reduce the communication complexity, we propo…
Distributed OptimizationQuantization