Improving Language Modeling using Densely Connected Recurrent Neural Networks
In this paper, we introduce the novel concept of densely connected layers into recurrent neural networks. We evaluate our proposed architecture on the Penn Treebank language modeling task. We show that we can obtain similar perplexity scores with six times fewer parameters compared to a standard stacked 2-layer LSTM model trained with dropout (Zaremba et al. 2014). In contrast with the current usage of skip connections, we show that densely connecting only a few stacked layers with skip connections already yields significant perplexity reductions.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information
Sentence matching is widely used in various natural language tasks such as natural language inference, paraphrase identification, and question answering. For these tasks, understanding logical and semantic relationship b…
Natural Language InferenceParaphrase IdentificationQuestion AnsweringSentenceDensely Connected Attention Propagation for Reading Comprehension
We propose DecaProp (Densely Connected Attention Propagation), a new densely connected neural architecture for reading comprehension (RC). There are two distinct characteristics of our model. Firstly, our model densely c…
AllOpen-Domain Question AnsweringQuestion AnsweringReading ComprehensionCharacter-Level Feature Extraction with Densely Connected Networks
Generating character-level features is an important step for achieving good results in various natural language processing tasks. To alleviate the need for human labor in generating hand-crafted features, methods that ut…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NER+3Mathematical Modeling and Convergence Analysis of Deep Neural Networks with Dense Layer Connectivities in Deep Learning
In deep learning, dense layer connectivity has become a key design principle in deep neural networks (DNNs), enabling efficient information flow and strong performance across a range of applications. In this work, we mod…
A Temporal Densely Connected Recurrent Network for Event-based Human Pose Estimation
Event camera is an emerging bio-inspired vision sensors that report per-pixel brightness changes asynchronously. It holds noticeable advantage of high dynamic range, high speed response, and low power budget that enable …
Pose Estimation