Performance Optimization on Model Synchronization in Parallel Stochastic Gradient Descent Based SVM

2019-05-03 · Vibhatha Abeykoon, Geoffrey Fox, Minje Kim

Understanding the bottlenecks in implementing stochastic gradient descent (SGD)-based distributed support vector machines (SVM) algorithm is important in training larger data sets. The communication time to do the model synchronization across the parallel processes is the main bottleneck that causes inefficiency in the training process. The model synchronization is directly affected by the mini-batch size of data processed before the global synchronization. In producing an efficient distributed model, the communication time in training model synchronization has to be as minimum as possible while retaining a high testing accuracy. The effect from model synchronization frequency over the convergence of the algorithm and accuracy of the generated model must be well understood to design an efficient distributed model. In this research, we identify the bottlenecks in model synchronization in parallel stochastic gradient descent (PSGD)-based SVM algorithm with respect to the training model synchronization frequency (MSF). Our research shows that by optimizing the MSF in the data sets that we used, a reduction of 98\% in communication time can be gained (16x - 24x speed up) with respect to high-frequency model synchronization. The training model optimization discussed in this paper guarantees a higher accuracy than the sequential algorithm along with faster convergence.

📄 PDF Abstract BibTeX arXiv:1905.01219

Code (0)

등록된 구현이 없습니다.

Tasks

Model Optimization

Methods 이 논문이 사용한 방법론

SPEED The monocular depth estimation (MDE) is the task of estimating depth from a single frame. This information is an essential knowledge in many computer vision tasks such as scene…

SVM A Support Vector Machine, or SVM, is a non-parametric supervised learning model. For non-linear classification and regression, they utilise the kernel trick to map inputs…

HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

2011-06-28 · Feng Niu, Benjamin Recht, Christopher Re, Stephen J. Wright

Stochastic Gradient Descent (SGD) is a popular algorithm that can achieve state-of-the-art performance on a variety of machine learning tasks. Several researchers have recently proposed schemes to parallelize SGD, but al…

Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent

2011-12-01 · NeurIPS 2011 12 · Benjamin Recht, Christopher Re, Stephen Wright, Feng Niu

Silent Inconsistency in Data-Parallel Full Fine-Tuning: Diagnosing Worker-Level Optimization Misalignment

2026-02-16 · Hong Li, Zhen Zhou, Honggang Zhang, Yuping Luo 외 arxiv

Data-parallel (DP) training with synchronous all-reduce is a dominant paradigm for full-parameter fine-tuning of large language models (LLMs). While parameter synchronization guarantees numerical equivalence of model wei…

Learning Efficient Parameter Server Synchronization Policies for Distributed SGD

2020-05-01 · ICLR 2020 1 · Rong Zhu, Sheng Yang, Andreas Pfadler, Zhengping Qian 외

We apply a reinforcement learning (RL) based approach to learning optimal synchronization policies used for Parameter Server-based distributed training of machine learning models with Stochastic Gradient Descent (SGD). U…

Q-Learning Reinforcement Learning (RL)

SWIFT: Rapid Decentralized Federated Learning via Wait-Free Model Communication

2022-10-25 · Marco Bornstein, Tahseen Rabbani, Evan Wang, Amrit Singh Bedi 외

The decentralized Federated Learning (FL) setting avoids the role of a potentially unreliable or untrustworthy central host by utilizing groups of clients to collaboratively train a model via localized training and model…

Federated Learning image-classification Image Classification