Going Wider: Recurrent Neural Network With Parallel Cells
Recurrent Neural Network (RNN) has been widely applied for sequence modeling. In RNN, the hidden states at current step are full connected to those at previous step, thus the influence from less related features at previous step may potentially decrease model's learning ability. We propose a simple technique called parallel cells (PCs) to enhance the learning ability of Recurrent Neural Network (RNN). In each layer, we run multiple small RNN cells rather than one single large cell. In this paper, we evaluate PCs on 2 tasks. On language modeling task on PTB (Penn Tree Bank), our model outperforms state of art models by decreasing perplexity from 78.6 to 75.3. On Chinese-English translation task, our model increases BLEU score for 0.39 points than baseline model.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingTranslationSimilar Papers 제목 키워드 기반
Dilated Recurrent Neural Networks
Learning with recurrent neural networks (RNNs) on long sequences is a notoriously difficult task. There are three major challenges: 1) complex dependencies, 2) vanishing and exploding gradients, and 3) efficient parallel…
Sequential Image ClassificationGo Wider Instead of Deeper
More transformer blocks with residual connections have recently achieved impressive results on various tasks. To achieve better performance with fewer trainable parameters, recent methods are proposed to go shallower by …
Image ClassificationMixture-of-ExpertsTime Matters: Time-Aware LSTMs for Predictive Business Process Monitoring
Predictive business process monitoring (PBPM) aims to predict future process behavior during ongoing process executions based on event log data. Especially, techniques for the next activity and timestamp prediction can h…
Parallel Long Short-Term Memory for Multi-stream Classification
Recently, machine learning methods have provided a broad spectrum of original and efficient algorithms based on Deep Neural Networks (DNN) to automatically predict an outcome with respect to a sequence of inputs. Recurre…
ClassificationGeneral ClassificationThick-Net: Parallel Network Structure for Sequential Modeling
Recurrent neural networks have been widely used in sequence learning tasks. In previous studies, the performance of the model has always been improved by either wider or deeper structures. However, the former becomes mor…
Language ModelingLanguage Modellingtext-classificationText Classification