A Robust Multi-Batch L-BFGS Method for Machine Learning
This paper describes an implementation of the L-BFGS method designed to deal with two adversarial situations. The first occurs in distributed computing environments where some of the computational nodes devoted to the evaluation of the function and gradient are unable to return results on time. A similar challenge occurs in a multi-batch approach in which the data points used to compute function and gradients are purposely changed at each iteration to accelerate the learning process. Difficulties arise because L-BFGS employs gradient differences to update the Hessian approximations, and when these gradients are computed using different data points the updating process can be unstable. This paper shows how to perform stable quasi-Newton updating in the multi-batch setting, studies the convergence properties for both convex and nonconvex functions, and illustrates the behavior of the algorithm in a distributed computing platform on binary classification logistic regression and neural network training problems that arise in machine learning.
Code (0)
등록된 구현이 없습니다.
Tasks
BIG-bench Machine LearningBinary ClassificationDistributed ComputingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Progressive Batching L-BFGS Method for Machine Learning
The standard L-BFGS method relies on gradient approximations that are not dominated by noise, so that search directions are descent directions, the line search is reliable, and quasi-Newton updating yields useful quadrat…
BIG-bench Machine LearningA Multi-Batch L-BFGS Method for Machine Learning
The question of how to parallelize the stochastic gradient descent (SGD) method has received much attention in the literature. In this paper, we focus instead on batch methods that use a sizeable fraction of the training…
BIG-bench Machine LearningDistributed ComputingAn Adaptive Memory Multi-Batch L-BFGS Algorithm for Neural Network Training
Motivated by the potential for parallel implementation of batch-based algorithms and the accelerated convergence achievable with approximated second order information a limited memory version of the BFGS algorithm has be…
On the Acceleration of L-BFGS with Second-Order Information and Stochastic Batches
This paper proposes a framework of L-BFGS based on the (approximate) second-order information with stochastic batches, as a novel approach to the finite-sum minimization problems. Different from the classical L-BFGS wher…
Large-scale L-BFGS using MapReduce
L-BFGS has been applied as an effective parameter estimation method for various machine learning algorithms since 1980s. With an increasing demand to deal with massive instances and variables, it is important to scale up…
BIG-bench Machine Learningparameter estimation