Short-term Memory of Deep RNN
The extension of deep learning towards temporal data processing is gaining an increasing research interest. In this paper we investigate the properties of state dynamics developed in successive levels of deep recurrent neural networks (RNNs) in terms of short-term memory abilities. Our results reveal interesting insights that shed light on the nature of layering as a factor of RNN design. Noticeably, higher layers in a hierarchically organized RNN architecture results to be inherently biased towards longer memory spans even prior to training of the recurrent connections. Moreover, in the context of Reservoir Computing framework, our analysis also points out the benefit of a layered recurrent organization as an efficient approach to improve the memory skills of reservoir models.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Transfer between long-term and short-term memory using Conceptors
We introduce a recurrent neural network model of working memory combining short-term and long-term components. e short-term component is modelled using a gated reservoir model that is trained to hold a value from an inpu…
Predicting the Transition from Short-term to Long-term Memory based on Deep Neural Network
Memory is an essential element in people's daily life based on experience. So far, many studies have analyzed electroencephalogram (EEG) signals at encoding to predict later remembered items, but few studies have predict…
EEGElectroencephalogram (EEG)RetrievalImprove Long-term Memory Learning Through Rescaling the Error Temporally
This paper studies the error metric selection for long-term memory learning in sequence modelling. We examine the bias towards short-term memory in commonly used errors, including mean absolute/squared error. Our finding…
MemoNav: Selecting Informative Memories for Visual Navigation
Image-goal navigation is a challenging task, as it requires the agent to navigate to a target indicated by an image in a previously unseen scene. Current methods introduce diverse memory mechanisms which save navigation …
Action GenerationGraph AttentionNavigateVisual NavigationFast Weight Long Short-Term Memory
Associative memory using fast weights is a short-term memory mechanism that substantially improves the memory capacity and time scale of recurrent neural networks (RNNs). As recent studies introduced fast weights only to…
Retrieval