Visual Grounding via Accumulated Attention
Visual Grounding (VG) aims to locate the most relevant object or region in an image, based on a natural language query. The query can be a phrase, a sentence or even a multi-round dialogue. There are three main challenges in VG: 1) what is the main focus in a query; 2) how to understand an image; 3) how to locate an object. Most existing methods combine all the information curtly, which may suffer from the problem of information redundancy (i.e. ambiguous query, complicated image and a large number of objects). In this paper, we formulate these challenges as three attention problems and propose an accumulated attention (A-ATT) mechanism to reason among them jointly. Our A-ATT mechanism can circularly accumulate the attention for useful information in image, query, and objects, while the noises are ignored gradually. We evaluate the performance of A-ATT on four popular datasets (namely ReferCOCO, ReferCOCO+, ReferCOCOg, and Guesswhat?!), and the experimental results show the superiority of the proposed method in term of accuracy.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceVisual GroundingSimilar Papers 제목 키워드 기반
Learning to Assemble Neural Module Tree Networks for Visual Grounding
Visual grounding, a task to ground (i.e., localize) natural language in images, essentially requires composite visual reasoning. However, existing methods over-simplify the composite nature of language into a monolithic …
Dependency ParsingNatural Language Visual GroundingSentenceSentence Embedding+4Learning to Compose and Reason with Language Tree Structures for Visual Grounding
Grounding natural language in images, such as localizing "the black dog on the left of the tree", is one of the core problems in artificial intelligence, as it needs to comprehend the fine-grained and compositional langu…
Visual GroundingVisual ReasoningDynamic Accumulated Attention Map for Interpreting Evolution of Decision-Making in Vision Transformer
Various Vision Transformer (ViT) models have been widely used for image recognition tasks. However, existing visual explanation methods can not display the attention flow hidden inside the inner structure of ViT models, …
Decision MakingYour Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding
Visual grounding seeks to localize the image region corresponding to a free-form text description. Recently, the strong multimodal capabilities of Large Vision-Language Models (LVLMs) have driven substantial improvements…
Language ModelingLanguage ModellingVisual GroundingWord2Pix: Word to Pixel Cross Attention Transformer in Visual Grounding
Current one-stage methods for visual grounding encode the language query as one holistic sentence embedding before fusion with visual feature. Such a formulation does not treat each word of a query sentence on par when m…
DecoderSentenceSentence EmbeddingSentence-Embedding+1