Unsupervised Image Captioning
Deep neural networks have achieved great successes on the image captioning task. However, most of the existing models depend heavily on paired image-sentence datasets, which are very expensive to acquire. In this paper, we make the first attempt to train an image captioning model in an unsupervised manner. Instead of relying on manually labeled image-sentence pairs, our proposed model merely requires an image set, a sentence corpus, and an existing visual concept detector. The sentence corpus is used to teach the captioning model how to generate plausible sentences. Meanwhile, the knowledge in the visual concept detector is distilled into the captioning model to guide the model to recognize the visual concepts in an image. In order to further encourage the generated captions to be semantically consistent with the image, the image and caption are projected into a common latent space so that they can reconstruct each other. Given that the existing sentence corpora are mainly designed for linguistic research and are thus with little reference to image contents, we crawl a large-scale image description corpus of two million natural sentences to facilitate the unsupervised image captioning scenario. Experimental results show that our proposed model is able to produce quite promising results without any caption annotations.
Code (1)
Tasks
Image CaptioningImage DescriptionSentenceSimilar Papers 제목 키워드 기반
Emergent Natural Language with Communication Games for Improving Image Captioning Capabilities without Additional Data
Image captioning is an important problem in developing various AI systems, and these tasks require large volumes of annotated images to train the models. Since all existing labelled datasets are already used for training…
Multi-agent Reinforcement LearningImage CaptioningObject-Centric Unsupervised Image Captioning
Image captioning is a longstanding problem in the field of computer vision and natural language processing. To date, researchers have produced impressive state-of-the-art performance in the age of deep learning. Most of …
Image CaptioningObjectSentenceZero-shot Image Captioning by Anchor-augmented Vision-Language Space Alignment
CLIP (Contrastive Language-Image Pre-Training) has shown remarkable zero-shot transfer capabilities in cross-modal correlation tasks such as visual classification and image retrieval. However, its performance in cross-mo…
Computational EfficiencyImage CaptioningImage RetrievalRetrievalVLRM: Vision-Language Models act as Reward Models for Image Captioning
In this work, we present an unsupervised method for enhancing an image captioning model (in our case, BLIP2) using reinforcement learning and vision-language models like CLIP and BLIP2-ITM as reward models. The RL-tuned …
Image Captioningreinforcement-learningReinforcement LearningUnpaired Image Captioning via Scene Graph Alignments
Most of current image captioning models heavily rely on paired image-caption datasets. However, getting large scale image-caption paired data is labor-intensive and time-consuming. In this paper, we present a scene graph…
DecoderImage CaptioningSentence