Learning Longer-term Dependencies via Grouped Distributor Unit
Learning long-term dependencies still remains difficult for recurrent neural networks (RNNs) despite their success in sequence modeling recently. In this paper, we propose a novel gated RNN structure, which contains only one gate. Hidden states in the proposed grouped distributor unit (GDU) are partitioned into groups. For each group, the proportion of memory to be overwritten in each state transition is limited to a constant and is adaptively distributed to each group member. In other word, every separate group has a fixed overall update rate, yet all units are allowed to have different paces. Information is therefore forced to be latched in a flexible way, which helps the model to capture long-term dependencies in data. Besides having a simpler structure, GDU is demonstrated experimentally to outperform LSTM and GRU on tasks including both pathological problems and natural data set.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Public Good Provision with a Distributor
We present a model of public good provision with a distributor. Our main result describes a symmetric mixed-strategy equilibrium, where all agents contribute to a common fund with probability $p$ and the distributor prov…
AllRedistributor: Transforming Empirical Data Distributions
We present an algorithm and package, Redistributor, which forces a collection of scalar samples to follow a desired distribution. When given independent and identically distributed samples of some random variable $S$ and…
Style TransferRecurrent Highway Networks with Grouped Auxiliary Memory
Recurrent neural networks (RNNs) are challenging to train, let alone those with deep spatial structures. Architectures built upon highway connections such as Recurrent Highway Network (RHN) were developed to allow larger…
image-classificationImage ClassificationLanguage ModelingLanguage Modelling+2Predicting Electricity Consumption using Deep Recurrent Neural Networks
Electricity consumption has increased exponentially during the past few decades. This increase is heavily burdening the electricity distributors. Therefore, predicting the future demand for electricity consumption will p…
Long History Short-Term Memory for Long-Term Video Prediction
While video prediction approaches have advanced considerably in recent years, learning to predict long-term future is challenging — ambiguous future or error propagation over time yield blurry predictions. To address thi…
Video Prediction