Encoding-based Memory Modules for Recurrent Neural Networks
Learning to solve sequential tasks with recurrent models requires the ability to memorize long sequences and to extract task-relevant features from them. In this paper, we study the memorization subtask from the point of view of the design and training of recurrent neural networks. We propose a new model, the Linear Memory Network, which features an encoding-based memorization component built with a linear autoencoder for sequences. We extend the memorization component with a modular memory that encodes the hidden state sequence at different sampling frequencies. Additionally, we provide a specialized training algorithm that initializes the memory to efficiently encode the hidden activations of the network. The experimental results on synthetic and real-world datasets show that specializing the training algorithm to train the memorization component always improves the final performance whenever the memorization of long sequences is necessary to solve the problem.
Code (0)
등록된 구현이 없습니다.
Tasks
MemorizationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Unidirectional Video Denoising by Mimicking Backward Recurrent Modules with Look-ahead Forward Ones
While significant progress has been made in deep video denoising, it remains very challenging for exploiting historical and future frames. Bidirectional recurrent networks (BiRNN) have exhibited appealing performance in …
DenoisingVideo DenoisingVideo RestorationGated Recurrent Unit Based Acoustic Modeling with Future Context
The use of future contextual information is typically shown to be helpful for acoustic modeling. However, for the recurrent neural network (RNN), it's not so easy to model the future temporal context effectively, meanwhi…
Learning to update Auto-associative Memory in Recurrent Neural Networks for Improving Sequence Memorization
Learning to remember long sequences remains a challenging task for recurrent neural networks. Register memory and attention mechanisms were both proposed to resolve the issue with either high computational cost to retain…
MemorizationRepresentation LearningPartially Non-Recurrent Controllers for Memory-Augmented Neural Networks
Memory-Augmented Neural Networks (MANNs) are a class of neural networks equipped with an external memory, and are reported to be effective for tasks requiring a large long-term memory and its selective use. The core modu…
Reservoir Computing for Fast, Simplified Reinforcement Learning on Memory Tasks
Tasks in which rewards depend upon past information not available in the current observation set can only be solved by agents that are equipped with short-term memory. Usual choices for memory modules include trainable r…
DecoderMeta-Learningreinforcement-learningReinforcement Learning