Memory-augmented Attention Modelling for Videos
We present a method to improve video description generation by modeling higher-order interactions between video frames and described concepts. By storing past visual attention in the video associated to previously generated words, the system is able to decide what to look at and describe in light of what it has already looked at and described. This enables not only more effective local attention, but tractable consideration of the video sequence while generating each word. Evaluation on the challenging and popular MSVD and Charades datasets demonstrates that the proposed architecture outperforms previous video description approaches without requiring external temporal video features.
Code (1)
Tasks
Video DescriptionSimilar Papers 제목 키워드 기반
Do Transformers Need Deep Long-Range Memory
Deep attention models have advanced the modelling of sequential data across many domains. For language modelling in particular, the Transformer-XL -- a Transformer augmented with a long-range memory of past activations -…
Deep AttentionLanguage ModellingDo Transformers Need Deep Long-Range Memory?
Deep attention models have advanced the modelling of sequential data across many domains. For language modelling in particular, the Transformer-XL {---} a Transformer augmented with a long-range memory of past activation…
Deep AttentionLanguage ModellingM3: Multimodal Memory Modelling for Video Captioning
Video captioning which automatically translates video clips into natural language sentences is a very important task in computer vision. By virtue of recent deep learning technologies, video captioning has made great pro…
SentenceVideo CaptioningMemory-Augmented Non-Local Attention for Video Super-Resolution
In this paper, we propose a novel video super-resolution method that aims at generating high-fidelity high-resolution (HR) videos from low-resolution (LR) ones. Previous methods predominantly leverage temporal neighbor f…
Analog Video RestorationSuper-ResolutionVideo Super-ResolutionMemory-augmented conformer for improved end-to-end long-form ASR
Conformers have recently been proposed as a promising modelling approach for automatic speech recognition (ASR), outperforming recurrent neural network-based approaches and transformers. Nevertheless, in general, the per…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)DecoderForm+2