Mind's Eye: A Recurrent Visual Representation for Image Caption Generation
In this paper we explore the bi-directional mapping between images and their sentence-based descriptions. Critical to our approach is a recurrent neural network that attempts to dynamically build a visual representation of the scene as a caption is being generated or read. The representation automatically learns to remember long-term visual concepts. Our model is capable of both generating novel captions given an image, and reconstructing visual features given an image description. We evaluate our approach on several tasks. These include sentence generation, sentence retrieval and image retrieval. State-of-the-art results are shown for the task of generating novel image descriptions. When compared to human generated captions, our automatically generated captions are equal to or preferred by humans $21.0\%$ of the time. Results are better than or comparable to state-of-the-art results on the image and sentence retrieval tasks for methods using similar visual features.
Code (0)
등록된 구현이 없습니다.
Tasks
Caption GenerationImage DescriptionImage RetrievalRetrievalSentenceSentence RetrievalSimilar Papers 제목 키워드 기반
HorNet: A Hierarchical Offshoot Recurrent Network for Improving Person Re-ID via Image Captioning
Person re-identification (re-ID) aims to recognize a person-of-interest across different cameras with notable appearance variance. Existing research works focused on the capability and robustness of visual representation…
Generative Adversarial NetworkImage CaptioningPerson Re-IdentificationMindSemantix: Deciphering Brain Visual Experiences with a Brain-Language Model
Deciphering the human visual experience through brain activities captured by fMRI represents a compelling and cutting-edge challenge in the field of neuroscience research. Compared to merely predicting the viewed image i…
Brain DecodingLanguage ModelingLanguage ModellingSelf-Supervised LearningLearning a Recurrent Visual Representation for Image Caption Generation
In this paper we explore the bi-directional mapping between images and their sentence-based descriptions. We propose learning this mapping using a recurrent neural network. Unlike previous approaches that map both senten…
Caption GenerationImage RetrievalRetrievalSentence+1Recurrent Image Captioner: Describing Images with Spatial-Invariant Transformation and Attention Filtering
Along with the prosperity of recurrent neural network in modelling sequential data and the power of attention mechanism in automatically identify salient information, image captioning, a.k.a., image description, has been…
DecoderImage CaptioningImage DescriptionVariational InferenceControlled Caption Generation for Images Through Adversarial Attacks
Deep learning is found to be vulnerable to adversarial examples. However, its adversarial susceptibility in image caption generation is under-explored. We study adversarial examples for vision and language models, which …
Caption GenerationImage CaptioningLanguage Modelling