Distributed Parameter Map-Reduce

2015-10-03 · Qi Li

This paper describes how to convert a machine learning problem into a series of map-reduce tasks. We study logistic regression algorithm. In logistic regression algorithm, it is assumed that samples are independent and each sample is assigned a probability. Parameters are obtained by maxmizing the product of all sample probabilities. Rapid expansion of training samples brings challenges to machine learning method. Training samples are so many that they can be only stored in distributed file system and driven by map-reduce style programs. The main step of logistic regression is inference. According to map-reduce spirit, each sample makes inference through a separate map procedure. But the premise of inference is that the map procedure holds parameters for all features in the sample. In this paper, we propose Distributed Parameter Map-Reduce, in which not only samples, but also parameters are distributed in nodes of distributed filesystem. Through a series of map-reduce tasks, we assign each sample parameters for its features, make inference for the sample and update paramters of the model. The above processes are excuted looply until convergence. We test the proposed algorithm in actual hadoop production environment. Experiments show that the acceleration of the algorithm is in linear relationship with the number of cluster nodes.

📄 PDF Abstract BibTeX arXiv:1510.00817

Code (0)

등록된 구현이 없습니다.

Tasks

BIG-bench Machine Learning regression

Methods 이 논문이 사용한 방법론

Logistic Regression Logistic Regression, despite its name, is a linear model for classification rather than regression. Logistic regression is also known in the literature as logit regression,…

Distributed Low-Rank Estimation Based on Joint Iterative Optimization in Wireless Sensor Networks

2014-11-05 · S. Xu, R. C. de Lamare, H. V. Poor

This paper proposes a novel distributed reduced--rank scheme and an adaptive algorithm for distributed estimation in wireless sensor networks. The proposed distributed scheme is based on a transformation that performs di…

Dimensionality Reduction

Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training

2018-11-08 · NeurIPS 2018 12 · Youjie Li, Mingchao Yu, Songze Li, Salman Avestimehr 외

Distributed training of deep nets is an important technique to address some of the present day computing challenges like memory consumption and computational demands. Classical distributed approaches, synchronous or asyn…

GPU

Reduced Network Cumulative Constraint Violation for Distributed Bandit Convex Optimization under Slater Condition

2024-11-18 · Kunpeng Zhang, Xinlei Yi, Jinliang Ding, Ming Cao 외

This paper studies the distributed bandit convex optimization problem with time-varying inequality constraints, where the goal is to minimize network regret and cumulative constraint violation. To calculate network cumul…

Dynamic Parameter Allocation in Parameter Servers

2020-02-03 · Alexander Renz-Wieland, Rainer Gemulla, Steffen Zeuch, Volker Markl

To keep up with increasing dataset sizes and model complexity, distributed training has become a necessity for large machine learning tasks. Parameter servers ease the implementation of distributed parameter management--…

BIG-bench Machine Learning Management

Efficient All-reduce for Distributed DNN Training in Optical Interconnect System

2022-07-22 · Fei Dai, Yawen Chen, Zhiyi Huang, Haibo Zhang 외

Communication efficiency plays an important role in accelerating the distributed training of Deep Neural Networks (DNN). All-reduce is the crucial communication primitive to reduce model parameters in distributed DNN tra…

All