Recurrent Neural Network Regularization
We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful technique for regularizing neural networks, does not work well with RNNs and LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show that it substantially reduces overfitting on a variety of tasks. These tasks include language modeling, speech recognition, image caption generation, and machine translation.
Code (21)
Tasks
Caption GenerationImage CaptioningLanguage ModelingLanguage ModellingMachine TranslationSpeech RecognitionTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Non-local Recurrent Regularization Networks for Multi-view Stereo
In deep multi-view stereo networks, cost regularization is crucial to achieve accurate depth estimation. Since 3D cost volume filtering is usually memory-consuming, recurrent 2D cost map regularization has recently becom…
Depth EstimationVariational Inference-Based Dropout in Recurrent Neural Networks for Slot Filling in Spoken Language Understanding
This paper proposes to generalize the variational recurrent neural network (RNN) with variational inference (VI)-based dropout regularization employed for the long short-term memory (LSTM) cells to more advanced RNN arch…
slot-fillingSlot FillingSpoken Language UnderstandingVariational InferenceRecurrent DNNs and its Ensembles on the TIMIT Phone Recognition Task
In this paper, we have investigated recurrent deep neural networks (DNNs) in combination with regularization techniques as dropout, zoneout, and regularization post-layer. As a benchmark, we chose the TIMIT phone recogni…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)speech-recognitionSpeech RecognitionRevisiting Activation Regularization for Language RNNs
Recurrent neural networks (RNNs) serve as a fundamental building block for many sequence tasks across natural language processing. Recent research has focused on recurrent dropout techniques or custom RNN cells in order …
L2 RegularizationLanguage ModelingLanguage ModellingRecurrent Neural Network Training with Convex Loss and Regularization Functions by Extended Kalman Filtering
This paper investigates the use of extended Kalman filtering to train recurrent neural networks with rather general convex loss functions and regularization terms on the network parameters, including $\ell_1$-regularizat…
Model Predictive Control