Accelerating recurrent neural network training using sequence bucketing and multi-GPU data parallelization
An efficient algorithm for recurrent neural network training is presented. The approach increases the training speed for tasks where a length of the input sequence may vary significantly. The proposed approach is based on the optimal batch bucketing by input sequence length and data parallelization on multiple graphical processing units. The baseline training performance without sequence bucketing is compared with the proposed solution for a different number of buckets. An example is given for the online handwriting recognition task using an LSTM recurrent neural network. The evaluation is performed in terms of the wall clock time, number of epochs, and validation loss value.
Code (0)
등록된 구현이 없습니다.
Tasks
GPUHandwriting RecognitionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A comprehensive study of batch construction strategies for recurrent neural networks in MXNet
In this work we compare different batch construction methods for mini-batch training of recurrent neural networks. While popular implementations like TensorFlow and MXNet suggest a bucketing approach to improve the paral…
Noisy Speech Recognitionspeech-recognitionSpeech RecognitionOrdinal Bucketing for Game Trees using Dynamic Quantile Approximation
In this paper, we present a simple and cheap ordinal bucketing algorithm that approximately generates $q$-quantiles from an incremental data stream. The bucketing is done dynamically in the sense that the amount of bucke…
RecLight: A Recurrent Neural Network Accelerator with Integrated Silicon Photonics
Recurrent Neural Networks (RNNs) are used in applications that learn dependencies in data sequences, such as speech recognition, human activity recognition, and anomaly detection. In recent years, newer RNN variants, suc…
Activity RecognitionAnomaly DetectionHuman Activity Recognitionspeech-recognition+1Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing
In Byzantine robust distributed or federated learning, a central server wants to train a machine learning model over data distributed across multiple workers. However, a fraction of these workers may deviate from the pre…
Distributed OptimizationFederated LearningLearning Recurrent Binary/Ternary Weights
Recurrent neural networks (RNNs) have shown excellent performance in processing sequence data. However, they are both complex and memory intensive due to their recursive nature. These limitations make RNNs difficult to e…
Language ModelingLanguage Modelling