Image Captioning with Unseen Objects
Image caption generation is a long standing and challenging problem at the intersection of computer vision and natural language processing. A number of recently proposed approaches utilize a fully supervised object recognition model within the captioning approach. Such models, however, tend to generate sentences which only consist of objects predicted by the recognition models, excluding instances of the classes without labelled training examples. In this paper, we propose a new challenging scenario that targets the image captioning problem in a fully zero-shot learning setting, where the goal is to be able to generate captions of test images containing objects that are not seen during training. The proposed approach jointly uses a novel zero-shot object detection model and a template-based sentence generator. Our experiments show promising results on the COCO dataset.
Code (0)
등록된 구현이 없습니다.
Tasks
Caption GenerationImage CaptioningObjectobject-detectionObject DetectionObject RecognitionSentenceZero-Shot LearningZero-Shot Object DetectionSimilar Papers 제목 키워드 기반
Learning to Select: A Fully Attentive Approach for Novel Object Captioning
Image captioning models have lately shown impressive results when applied to standard datasets. Switching to real-life scenarios, however, constitutes a challenge due to the larger variety of visual concepts which are no…
Image CaptioningLanguage ModelingLanguage ModellingCaptioning Images with Diverse Objects
Recent captioning models are limited in their ability to scale and describe concepts unseen in paired image-text corpora. We propose the Novel Object Captioner (NOC), a deep visual semantic captioning model that can desc…
ObjectObject RecognitionCaption Generation on Scenes with Seen and Unseen Object Categories
Image caption generation is one of the most challenging problems at the intersection of vision and language domains. In this work, we propose a realistic captioning task where the input scenes may incorporate visual obje…
Caption GenerationLanguage ModellingGuided Open Vocabulary Image Captioning with Constrained Beam Search
Existing image captioning models do not generalize well to out-of-domain images containing novel scenes or objects. This limitation severely hinders the use of these models in real world applications dealing with images …
Image CaptioningTAGWord EmbeddingsLearning Visual-Linguistic Adequacy, Fidelity, and Fluency for Novel Object Captioning
Novel object captioning (NOC) learns image captioning models for describing objects or visual concepts which are unseen (i.e., novel) in the training captions. Such captioning models need to sufficiently describe such vi…
Image Captioning