Batch-normalized Recurrent Highway Networks
Gradient control plays an important role in feed-forward networks applied to various computer vision tasks. Previous work has shown that Recurrent Highway Networks minimize the problem of vanishing or exploding gradients. They achieve this by setting the eigenvalues of the temporal Jacobian to 1 across the time steps. In this work, batch normalized recurrent highway networks are proposed to control the gradient flow in an improved way for network convergence. Specifically, the introduced model can be formed by batch normalizing the inputs at each recurrence loop. The proposed model is tested on an image captioning task using MSCOCO dataset. Experimental results indicate that the batch normalized recurrent highway networks converge faster and performs better compared with the traditional LSTM and RHN based models.
Code (1)
Tasks
Image CaptioningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Recurrent Batch Normalization
We propose a reparameterization of LSTM that brings the benefits of batch normalization to recurrent neural networks. Whereas previous works only apply batch normalization to the input-to-hidden transformation of RNNs, w…
General ClassificationLanguage ModelingLanguage ModellingQuestion Answering+2Batch Normalized Recurrent Neural Networks
Recurrent Neural Networks (RNNs) are powerful models for sequential data that have the potential to learn long-term dependencies. However, they are computationally expensive to train and difficult to parallelize. Recent …
Language Modellingspeech-recognitionSpeech RecognitionLearning text representation using recurrent convolutional neural network with highway layers
Recently, the rapid development of word embedding and neural networks has brought new inspiration to various NLP and IR tasks. In this paper, we describe a staged hybrid model combining Recurrent Convolutional Neural Net…
Sentiment AnalysisNeural Machine Translation with Recurrent Highway Networks
Recurrent Neural Networks have lately gained a lot of popularity in language modelling tasks, especially in neural machine translation(NMT). Very recent NMT models are based on Encoder-Decoder, where a deep LSTM based en…
DecoderLanguage ModellingMachine TranslationNMT+2Learning Multi-Level Information for Dialogue Response Selection by Highway Recurrent Transformer
With the increasing research interest in dialogue response generation, there is an emerging branch formulating this task as selecting next sentences, where given the partial dialogue contexts, the goal is to determine th…
Response GenerationSentence