Temporal Attention Modules for Memory-Augmented Neural Networks
We introduce two temporal attention modules which can be plugged into traditional memory augmented recurrent neural networks to improve their performance in natural language processing tasks. The temporal attention modules provide new inductive biases allowing the models to compute attention distributions over the different time steps of input sequences. The values of these attention distributions can be inspected to identify the sequence's elements that the model considered relevant during the inference. Using the Entity Network (Henaff et al., 2016) as the model backbone, experiments were made on the dataset bAbI tasks, a set of QA tasks. Due to the addition of the temporal attention modules, the performance metric increased 26% when the temporal attention was supervised, and 13,5% when it wasn't. Moreover, the usage of temporal attention modules proved useful at resolving reasoning tasks that the original model was unable to solve.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Streaming Attention-Based Models with Augmented Memory for End-to-End Speech Recognition
Attention-based models have been gaining popularity recently for their strong performance demonstrated in fields such as machine translation and automatic speech recognition. One major challenge of attention-based models…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Machine Translationspeech-recognition+2Attention based Memory video portrait matting
We proposed a novel trimap free video matting method based on the attention mechanism. By the nature of the problem, most existing approaches use either multiple computational expansive modules or complex algorithms to e…
Image MattingVideo MattingSRVP: Strong Recollection Video Prediction Model Using Attention-Based Spatiotemporal Correlation Fusion
Video prediction (VP) generates future frames by leveraging spatial representations and temporal context from past frames. Traditional recurrent neural network (RNN)-based models enhance memory cell structures to capture…
Video PredictionDAWN: Dual Augmented Memory Network for Unsupervised Video Object Tracking
Psychological studies have found that human visual tracking system involves learning, memory, and planning. Despite recent successes, not many works have focused on memory and planning in deep learning based tracking. We…
Object TrackingVideo Object TrackingVisual TrackingUnderstanding Memory Modules on Learning Simple Algorithms
Recent work has shown that memory modules are crucial for the generalization ability of neural networks on learning simple algorithms. However, we still have little understanding of the working mechanism of memory module…
Dimensionality Reduction