Interactive Key-Value Memory-augmented Attention for Image Paragraph Captioning
Image paragraph captioning (IPC) aims to generate a fine-grained paragraph to describe the visual content of an image. Significant progress has been made by deep neural networks, in which the attention mechanism plays an essential role. However, conventional attention mechanisms tend to ignore the past alignment information, which often results in problems of repetitive captioning and incomplete captioning. In this paper, we propose an Interactive key-value Memory- augmented Attention model for image Paragraph captioning (IMAP) to keep track of the attention history (salient objects coverage information) along with the update-chain of the decoder state and therefore avoid generating repetitive or incomplete image descriptions. In addition, we employ an adaptive attention mechanism to realize adaptive alignment from image regions to caption words, where an image region can be mapped to an arbitrary number of caption words while a caption word can also attend to an arbitrary number of image regions. Extensive experiments on a benchmark dataset (i.e., Stanford) demonstrate the effectiveness of our IMAP model.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderImage Paragraph CaptioningSimilar Papers 제목 키워드 기반
Frustratingly Short Attention Spans in Neural Language Modeling
Neural language models predict the next token using a latent representation of the immediate token history. Recently, various methods for augmenting neural language models with an attention mechanism over a differentiabl…
Language ModelingLanguage ModellingNeural Machine Translation with Key-Value Memory-Augmented Attention
Although attention-based Neural Machine Translation (NMT) has achieved remarkable progress in recent years, it still suffers from issues of repeating and dropping translations. To alleviate these issues, we propose a nov…
DecoderMachine TranslationNMTSentence+1Memory-augmented Dialogue Management for Task-oriented Dialogue Systems
Dialogue management (DM) decides the next action of a dialogue system according to the current dialogue state, and thus plays a central role in task-oriented dialogue systems. Since dialogue management requires to have a…
Dialogue ManagementManagementTask-Oriented Dialogue SystemsTemporal Attention Modules for Memory-Augmented Neural Networks
We introduce two temporal attention modules which can be plugged into traditional memory augmented recurrent neural networks to improve their performance in natural language processing tasks. The temporal attention modul…
Interactive Tracking: A Human-in-the-Loop Paradigm with Memory-Augmented Adaptation
Existing visual trackers mainly operate in a non-interactive, fire-and-forget manner, making them impractical for real-world scenarios that require human-in-the-loop adaptation. To overcome this limitation, we introduce …