Phrase-based Image Captioning
Generating a novel textual description of an image is an interesting problem that connects computer vision and natural language processing. In this paper, we present a simple model that is able to generate descriptive sentences given a sample image. This model has a strong focus on the syntax of the descriptions. We train a purely bilinear model that learns a metric between an image representation (generated from a previously trained Convolutional Neural Network) and phrases that are used to described them. The system is then able to infer phrases from a given image sample. Based on caption syntax statistics, we propose a simple language model that can produce relevant descriptions for a given test image using the phrases inferred. Our approach, which is considerably simpler than state-of-the-art models, achieves comparable results in two popular datasets for the task: Flickr30k and the recently proposed Microsoft COCO.
Code (0)
등록된 구현이 없습니다.
Tasks
DescriptiveImage CaptioningLanguage ModelingLanguage ModellingSimilar Papers 제목 키워드 기반
Visual Information Guided Zero-Shot Paraphrase Generation
Zero-shot paraphrase generation has drawn much attention as the large-scale high-quality paraphrase corpus is limited. Back-translation, also known as the pivot-based method, is typical to this end. Several works leverag…
DiversityImage CaptioningParaphrase GenerationTranslationSimilar Scenes arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning
Stylized image captioning systems aim to generate a caption not only semantically related to a given image but also consistent with a given style description. One of the biggest challenges with this task is the lack of s…
Data AugmentationImage CaptioningSentenceRefCaptioner: Multi-Reference Image-Grounded Video Captioning
Existing video captioning models generate natural descriptions of video content but cannot explicitly ground local visual elements to multiple reference images. We introduce multi-reference image-grounded video captionin…
Video ReconstructionVideo CaptioningPANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection
Intelligent systems that act in the world require image understanding that is both comprehensive and spatially grounded. Current vision-language models (VLMs) can generate fluent and detailed image captions, but reliably…
Panoptic SegmentationDense CaptioningPhrase GroundingGenerating Diverse and Descriptive Image Captions Using Visual Paraphrases
Recently there has been significant progress in image captioning with the help of deep learning. However, captions generated by current state-of-the-art models are still far from satisfactory, despite high scores in term…
DescriptiveDiversityImage Captioning