Learning to Guide Decoding for Image Captioning
Recently, much advance has been made in image captioning, and an encoder-decoder framework has achieved outstanding performance for this task. In this paper, we propose an extension of the encoder-decoder framework by adding a component called guiding network. The guiding network models the attribute properties of input images, and its output is leveraged to compose the input of the decoder at each time step. The guiding network can be plugged into the current encoder-decoder framework and trained in an end-to-end manner. Hence, the guiding vector can be adaptively learned according to the signal from the decoder, making itself to embed information from both image and language. Additionally, discriminative supervision can be employed to further improve the quality of guidance. The advantages of our proposed approach are verified by experiments carried out on the MS COCO dataset.
Code (0)
등록된 구현이 없습니다.
Tasks
AttributeDecoderImage CaptioningSimilar Papers 제목 키워드 기반
Fast Image Caption Generation with Position Alignment
Recent neural network models for image captioning usually employ an encoder-decoder architecture, where the decoder adopts a recursive sequence decoding way. However, such autoregressive decoding may result in sequential…
Caption GenerationDecoderImage CaptioningPosition+1AGIC: Attention-Guided Image Captioning to Improve Caption Relevance
Despite significant progress in image captioning, generating accurate and descriptive captions remains a long-standing challenge. In this study, we propose Attention-Guided Image Captioning (AGIC), which amplifies salien…
Image CaptioningGuiding Image Captioning Models Toward More Specific Captions
Image captioning is conventionally formulated as the task of generating captions for images that match the distribution of reference image-caption pairs. However, reference captions in standard captioning datasets are sh…
Image CaptioningImage RetrievalTransferable Decoding with Visual Entities for Zero-Shot Image Captioning
Image-to-text generation aims to describe images using natural language. Recently, zero-shot image captioning based on pre-trained vision-language models (VLMs) and large language models (LLMs) has made significant progr…
Caption GenerationHallucinationImage CaptioningImage to text+2Brain Captioning: Decoding human brain activity into images and text
Every day, the human brain processes an immense volume of visual information, relying on intricate neural mechanisms to perceive and interpret these stimuli. Recent breakthroughs in functional magnetic resonance imaging …
Brain DecodingDepth EstimationImage CaptioningImage Reconstruction+2