Hidden State Guidance: Improving Image Captioning using An Image Conditioned Autoencoder
Most RNN-based image captioning models receive supervision on the output words to mimic human captions. Therefore, the hidden states can only receive noisy gradient signals via layers of back-propagation through time, leading to less accurate generated captions. Consequently, we propose a novel framework, Hidden State Guidance (HSG), that matches the hidden states in the caption decoder to those in a teacher decoder trained on an easier task of autoencoding the captions conditioned on the image. During training with the REINFORCE algorithm, the conventional rewards are sentence-based evaluation metrics equally distributed to each generated word, no matter their relevance. HSG provides a word-level reward that helps the model learn better hidden representations. Experimental results demonstrate that HSG clearly outperforms various state-of-the-art caption decoders using either raw images or detected objects as inputs.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderImage CaptioningSentenceMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Guiding Image Captioning Models Toward More Specific Captions
Image captioning is conventionally formulated as the task of generating captions for images that match the distribution of reference image-caption pairs. However, reference captions in standard captioning datasets are sh…
Image CaptioningImage RetrievalShow, Tell and Discriminate: Image Captioning by Self-retrieval with Partially Labeled Data
The aim of image captioning is to generate captions by machine to describe image contents. Despite many efforts, generating discriminative captions for images remains non-trivial. Most traditional approaches imitate the …
Image CaptioningRetrievalProtect, Show, Attend and Tell: Empowering Image Captioning Models with Ownership Protection
By and large, existing Intellectual Property (IP) protection on deep neural networks typically i) focus on image classification task only, and ii) follow a standard digital watermarking framework that was conventionally …
Image Captioningimage-classificationImage ClassificationCgT-GAN: CLIP-guided Text GAN for Image Captioning
The large-scale visual-language pre-trained model, Contrastive Language-Image Pre-training (CLIP), has significantly improved image captioning for scenarios without human-annotated image-caption pairs. Recent advanced CL…
Image CaptioningImage Captioning with Context-Aware Auxiliary Guidance
Image captioning is a challenging computer vision task, which aims to generate a natural language description of an image. Most recent researches follow the encoder-decoder framework which depends heavily on the previous…
DecoderImage Captioning