Semi-Cyclic Stochastic Gradient Descent

2019-04-23 · Hubert Eichner, Tomer Koren, H. Brendan McMahan, Nathan Srebro, Kunal Talwar

We consider convex SGD updates with a block-cyclic structure, i.e. where each cycle consists of a small number of blocks, each with many samples from a possibly different, block-specific, distribution. This situation arises, e.g., in Federated Learning where the mobile devices available for updates at different times during the day have different characteristics. We show that such block-cyclic structure can significantly deteriorate the performance of SGD, but propose a simple approach that allows prediction with the same performance guarantees as for i.i.d., non-cyclic, sampling.

📄 PDF Abstract BibTeX arXiv:1904.10120

Code (0)

등록된 구현이 없습니다.

Tasks

Federated Learning

Methods 이 논문이 사용한 방법론

SGD Stochastic Gradient Descent is an iterative optimization technique that uses minibatches of data to form an expectation of the gradient, rather than the full gradient using…

Cyclic Block Coordinate Descent With Variance Reduction for Composite Nonconvex Optimization

2022-12-09 · Xufeng Cai, Chaobing Song, Stephen J. Wright, Jelena Diakonikolas

Nonconvex optimization is central in solving many machine learning problems, in which block-wise structure is commonly encountered. In this work, we propose cyclic block coordinate methods for nonconvex optimization prob…

Projected Semi-Stochastic Gradient Descent Method with Mini-Batch Scheme under Weak Strong Convexity Assumption

2016-12-16 · Jie Liu, Martin Takac

We propose a projected semi-stochastic gradient descent method with mini-batch for improving both the theoretical complexity and practical performance of the general stochastic gradient descent method (SGD). We are able …

BIG-bench Machine Learning

Stochastic Gradient Made Stable: A Manifold Propagation Approach for Large-Scale Optimization

2015-06-28 · Yadong Mu, Wei Liu, Wei Fan

Stochastic gradient descent (SGD) holds as a classical method to build large scale machine learning models over big data. A stochastic gradient is typically calculated from a limited number of samples (known as mini-batc…

Deep Reinforcement Learning using Cyclical Learning Rates

2020-07-31 · Ralf Gulde, Marc Tuscher, Akos Csiszar, Oliver Riedel 외

Deep Reinforcement Learning (DRL) methods often rely on the meticulous tuning of hyperparameters to successfully resolve problems. One of the most influential parameters in optimization procedures based on stochastic gra…

Deep Reinforcement Learning reinforcement-learning Reinforcement Learning Reinforcement Learning (RL)

Amplifying Rényi Differential Privacy via Shuffling

2019-07-11 · Eloïse Berthier, Sai Praneeth Karimireddy

Differential privacy is a useful tool to build machine learning models which do not release too much information about the training data. We study the R\'enyi differential privacy of stochastic gradient descent when each…