Guiding the Long-Short Term Memory Model for Image Caption Generation
In this work we focus on the problem of image caption generation. We propose an extension of the long short term memory (LSTM) model, which we coin gLSTM for short. In particular, we add semantic information extracted from the image as extra input to each unit of the LSTM block, with the aim of guiding the model towards solutions that are more tightly coupled to the image content. Additionally, we explore different length normalization strategies for beam search to avoid bias towards short sentences. On various benchmark datasets such as Flickr8K, Flickr30K and MS COCO, we obtain results that are on par with or better than the current state-of-the-art.
Code (0)
등록된 구현이 없습니다.
Tasks
Caption GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Guiding Long-Short Term Memory for Image Caption Generation
In this work we focus on the problem of image caption generation. We propose an extension of the long short term memory (LSTM) model, which we coin gLSTM for short. In particular, we add semantic information extracted fr…
Caption GenerationA sequential guiding network with attention for image captioning
The recent advances of deep learning in both computer vision (CV) and natural language processing (NLP) provide us a new way of understanding semantics, by which we can deal with more challenging tasks such as automatic …
DecoderImage CaptioningLearning From Failures: Efficient Reinforcement Learning Control with Episodic Memory
Reinforcement learning has achieved remarkable success in robot learning. However, under challenging exploration and contact-rich dynamics, early-stage training is frequently dominated by premature terminations such as c…
Reinforcement LearningMemoNav: Selecting Informative Memories for Visual Navigation
Image-goal navigation is a challenging task, as it requires the agent to navigate to a target indicated by an image in a previously unseen scene. Current methods introduce diverse memory mechanisms which save navigation …
Action GenerationGraph AttentionNavigateVisual NavigationReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries
Vision-language-action (VLA) models for closed-loop robot control are typically cast under the Markov assumption, making them prone to errors on tasks requiring historical context. To incorporate memory, existing VLAs ei…