Papers Distributed Optimization
“Distributed Optimization” 태그가 달린 논문 601편 · 필터 해제
New Bounds For Distributed Mean Estimation and Variance Reduction
We consider the problem of distributed mean estimation (DME), in which $n$ machines are each given a local $d$-dimensional vector $x_v \in \mathbb{R}^d$, and must cooperate to estimate the mean of their inputs $\mu = \fr…
Distributed OptimizationQuantizationAdaptive Sampling Distributed Stochastic Variance Reduced Gradient for Heterogeneous Distributed Datasets
We study distributed optimization algorithms for minimizing the average of \emph{heterogeneous} functions distributed across several machines with a focus on communication efficiency. In such settings, naively using the …
Distributed OptimizationThe Geometry of Sign Gradient Descent
Sign-based optimization methods have become popular in machine learning due to their favorable communication cost in distributed optimization and their surprisingly good performance in neural network training. Furthermor…
Distributed OptimizationDistributed Optimization over Block-Cyclic Data
We consider practical data characteristics underlying federated learning, where unbalanced and non-i.i.d. data from clients have a block-cyclic structure: each cycle contains several blocks, and each client's training da…
Distributed OptimizationFederated LearningIs Local SGD Better than Minibatch SGD?
We study local SGD (also known as parallel SGD and federated averaging), a natural and frequently used stochastic distributed optimization method. Its theoretical foundations are currently lacking and we highlight how al…
Distributed OptimizationDistributed Averaging Methods for Randomized Second Order Optimization
We consider distributed optimization problems where forming the Hessian is computationally challenging and communication is a significant bottleneck. We develop unbiased parameter averaging methods for randomized second …
Distributed OptimizationTraining Large Neural Networks with Constant Memory using a New Execution Algorithm
Widely popular transformer-based NLP models such as BERT and Turing-NLG have enormous capacity trending to billions of parameters. Current execution methods demand brute-force resources such as HBM devices and high speed…
CPUDistributed OptimizationNeural Architecture SearchDifferentially Quantized Gradient Methods
Consider the following distributed optimization scenario. A worker has access to training data that it uses to compute the gradients while a server decides when to stop iterative computation based on its target accuracy …
Distributed OptimizationQuantizationFedDANE: A Federated Newton-Type Method
Federated learning aims to jointly learn statistical models over massively distributed remote devices. In this work, we propose FedDANE, an optimization method that we adapt from DANE, a method for classical distributed …
Distributed OptimizationFederated LearningVocal Bursts Type PredictionManifold Identification for Ultimately Communication-Efficient Distributed Optimization
The expensive inter-machine communication is the bottleneck of distributed optimization. Existing study tackles this problem by shortening the communication rounds, but the reduction of per-round communication cost is no…
Distributed OptimizationAcceleration for Compressed Gradient Descent in Distributed Optimization
Due to the high communication cost in distributed and federated learning problems, methods relying on sparsification or quantization of communicated messages are becoming increasingly popular. While in other contexts the…
Distributed OptimizationFederated LearningQuantizationEstimating the Error of Randomized Newton Methods: A Bootstrap Approach
Randomized Newton methods have recently become the focus of intense research activity in large-scale and distributed optimization. Generally, these methods are based on a "computation-accuracy trade-off", which allows th…
Distributed OptimizationvalidGraph Learning Under Partial Observability
Many optimization, inference and learning tasks can be accomplished efficiently by means of decentralized processing algorithms where the network topology (i.e., the graph) plays a critical role in enabling the interacti…
Distributed OptimizationGraph LearningA Distributed Quasi-Newton Algorithm for Primal and Dual Regularized Empirical Risk Minimization
We propose a communication- and computation-efficient distributed optimization algorithm using second-order information for solving empirical risk minimization (ERM) problems with a nonsmooth regularization term. Our alg…
Distributed OptimizationQsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations
Communication bottleneck has been identified as a significant issue in distributed optimization of large-scale learning models. Recently, several approaches to mitigate this problem have been proposed, including differen…
Distributed OptimizationQuantizationOptimization for Reinforcement Learning: From Single Agent to Cooperative Agents
This article reviews recent advances in multi-agent reinforcement learning algorithms for large-scale control systems and communication networks, which learn to communicate and cooperate. We provide an overview of this e…
Distributed OptimizationMulti-agent Reinforcement Learningreinforcement-learningReinforcement Learning+1Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees
To reduce the long training time of large deep neural network (DNN) models, distributed synchronous stochastic gradient descent (S-SGD) is commonly used on a cluster of workers. However, the speedup brought by multiple w…
Distributed OptimizationGPUvqSGD: Vector Quantized Stochastic Gradient Descent
In this work, we present a family of vector quantization schemes \emph{vqSGD} (Vector-Quantized Stochastic Gradient Descent) that provide an asymptotic reduction in the communication cost with convergence guarantees in f…
Distributed OptimizationQuantizationLearning-Accelerated ADMM for Distributed Optimal Power Flow
We propose a novel data-driven method to accelerate the convergence of Alternating Direction Method of Multipliers (ADMM) for solving distributed DC optimal power flow (DC-OPF) where lines are shared between independent …
Distributed OptimizationOn the Convergence of Local Descent Methods in Federated Learning
In federated distributed learning, the goal is to optimize a global training objective defined over distributed devices, where the data shard at each device is sampled from a possibly different distribution (a.k.a., hete…
Distributed OptimizationFederated Learning