Image Captioning with Compositional Neural Module Networks
In image captioning where fluency is an important factor in evaluation, e.g., $n$-gram metrics, sequential models are commonly used; however, sequential models generally result in overgeneralized expressions that lack the details that may be present in an input image. Inspired by the idea of the compositional neural module networks in the visual question answering task, we introduce a hierarchical framework for image captioning that explores both compositionality and sequentiality of natural language. Our algorithm learns to compose a detail-rich sentence by selectively attending to different modules corresponding to unique aspects of each object detected in an input image to include specific descriptions such as counts and color. In a set of experiments on the MSCOCO dataset, the proposed model outperforms a state-of-the art model across multiple evaluation metrics, more importantly, presenting visually interpretable results. Furthermore, the breakdown of subcategories $f$-scores of the SPICE metric and human evaluation on Amazon Mechanical Turk show that our compositional module networks effectively generate accurate and detailed captions.
Code (0)
등록된 구현이 없습니다.
Tasks
Image CaptioningQuestion AnsweringSentenceVisual Question AnsweringVisual Question Answering (VQA)Similar Papers 제목 키워드 기반
Leveraging Textual Compositional Reasoning for Robust Change Captioning
Change captioning aims to describe changes between a pair of images. However, existing works rely on visual features alone, which often fail to capture subtle but meaningful changes because they lack the ability to repre…
Relational ReasoningFINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity
The advent of large Vision-Language Models (VLMs) has significantly advanced multimodal tasks, enabling more sophisticated and accurate reasoning across various applications, including image and video captioning, visual …
AttributeCross-Modal RetrievalImage CaptioningQuestion Answering+2A Neural Compositional Paradigm for Image Captioning
Mainstream captioning models often follow a sequential structure to generate captions, leading to issues such as introduction of irrelevant semantics, lack of diversity in the generated captions, and inadequate generaliz…
DiversityImage CaptioningA Neural Compositional Paradigm for Image Captioning
Mainstream captioning models often follow a sequential structure to generate cap- tions, leading to issues such as introduction of irrelevant semantics, lack of diversity in the generated captions, and inadequate general…
DiversityImage CaptioningThe Role of Syntactic Planning in Compositional Image Captioning
Image captioning has focused on generalizing to images drawn from the same distribution as the training set, and not to the more challenging problem of generalizing to different distributions of images. Recently, Nikolau…
Image Captioning