Deep Visual-Semantic Alignments for Generating Image Descriptions
We present a model that generates natural language descriptions of images and their regions. Our approach leverages datasets of images and their sentence descriptions to learn about the inter-modal correspondences between language and visual data. Our alignment model is based on a novel combination of Convolutional Neural Networks over image regions, bidirectional Recurrent Neural Networks over sentences, and a structured objective that aligns the two modalities through a multimodal embedding. We then describe a Multimodal Recurrent Neural Network architecture that uses the inferred alignments to learn to generate novel descriptions of image regions. We demonstrate that our alignment model produces state of the art results in retrieval experiments on Flickr8K, Flickr30K and MSCOCO datasets. We then show that the generated descriptions significantly outperform retrieval baselines on both full images and on a new dataset of region-level annotations.
Code (4)
Tasks
Cross-Modal RetrievalImage CaptioningImage-to-Text RetrievalRetrievalSentenceSimilar Papers 제목 키워드 기반
Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations
With the aim of promoting and understanding the multilingual version of image search, we leverage visual object detection and propose a model with diverse multi-head attention to learn grounded multilingual multimodal re…
DiversityImage Retrievalobject-detectionObject Detection+1Semantically-Prompted Language Models Improve Visual Descriptions
Language-vision models like CLIP have made significant strides in vision tasks, such as zero-shot image classification (ZSIC). However, generating specific and expressive visual descriptions remains challenging; descript…
ClassificationDescriptiveFine-Grained Image Classificationimage-classification+5SANDI: Story-and-Images Alignment
The Internet contains a multitude of social media posts and other of stories where text is interspersed with images. In these contexts, images are not simply used for general illustration, but are judiciously placed in c…
When an Image Tells a Story: The Role of Visual and Semantic Information for Generating Paragraph Descriptions
Generating multi-sentence image descriptions is a challenging task, which requires a good model to produce coherent and accurate paragraphs, describing salient objects in the image. We argue that multiple sources of info…
Image Paragraph CaptioningSentenceFine-Grained Image-Text Matching by Cross-Modal Hard Aligning Network
Current state-of-the-art image-text matching methods implicitly align the visual-semantic fragments, like regions in images and words in sentences, and adopt cross-attention mechanism to discover fine-grained cross-m…
Image-text matchingRetrievalSemantic correspondenceText Matching+2