Efficient Asynchronize Stochastic Gradient Algorithm with Structured Data
Deep learning has achieved impressive success in a variety of fields because of its good generalization. However, it has been a challenging problem to quickly train a neural network with a large number of layers. The existing works utilize the locality-sensitive hashing technique or some data structures on space partitioning to alleviate the training cost in each iteration. In this work, we try accelerating the computations in each iteration from the perspective of input data points. Specifically, for a two-layer fully connected neural network, when the training data have some special properties, e.g., Kronecker structure, each iteration can be completed in sublinear time in the data dimension.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Fully Distributed and Asynchronized Stochastic Gradient Descent for Networked Systems
This paper considers a general data-fitting problem over a networked system, in which many computing nodes are connected by an undirected graph. This kind of problem can find many real-world applications and has been stu…
Learning Fine-grained Image Similarity with Deep Ranking
Learning fine-grained image similarity is a challenging task. It needs to capture between-class and within-class image differences. This paper proposes a deep ranking model that employs deep learning techniques to learn …
General ClassificationTripletStochastic Iterative Hard Thresholding for Graph-structured Sparsity Optimization
Stochastic optimization algorithms update models with cheap per-iteration costs sequentially, which makes them amenable for large-scale data analysis. Such algorithms have been widely studied for structured sparse models…
Stochastic OptimizationStochastic Zeroth order Descent with Structured Directions
We introduce and analyze Structured Stochastic Zeroth order Descent (S-SZD), a finite difference approach that approximates a stochastic gradient on a set of $l\leq d$ orthogonal directions, where $d$ is the dimension of…
Quantum Algorithms for Structured Prediction
We introduce two quantum algorithms for solving structured prediction problems. We first show that a stochastic gradient descent that uses the quantum minimum finding algorithm and takes its probabilistic failure into ac…
BIG-bench Machine LearningPredictionQuantum Machine LearningStructured Prediction