Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing
Referring expression grounding aims at locating certain objects or persons in an image with a referring expression, where the key challenge is to comprehend and align various types of information from visual and textual domain, such as visual attributes, location and interactions with surrounding regions. Although the attention mechanism has been successfully applied for cross-modal alignments, previous attention models focus on only the most dominant features of both modalities, and neglect the fact that there could be multiple comprehensive textual-visual correspondences between images and referring expressions. To tackle this issue, we design a novel cross-modal attention-guided erasing approach, where we discard the most dominant information from either textual or visual domains to generate difficult training samples online, and to drive the model to discover complementary textual-visual correspondences. Extensive experiments demonstrate the effectiveness of our proposed method, which achieves state-of-the-art performance on three referring expression grounding datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Referring ExpressionSimilar Papers 제목 키워드 기반
Cross-Modal Relationship Inference for Grounding Referring Expressions
Grounding referring expressions is a fundamental yet challenging task facilitating human-machine communication in the physical world. It locates the target object in an image on the basis of the comprehension of the rela…
Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching
Referring expression grounding is an important and challenging task in computer vision. To avoid the laborious annotation in conventional referring grounding, unpaired referring grounding is introduced, where the trainin…
Image-text matchingReferring ExpressionText MatchingRelationship-Embedded Representation Learning for Grounding Referring Expressions
Grounding referring expressions in images aims to locate the object instance in an image described by a referring expression. It involves a joint understanding of natural language and image content, and is essential for …
Referring ExpressionRepresentation LearningCo-Grounding Networks with Semantic Attention for Referring Expression Comprehension in Videos
In this paper, we address the problem of referring expression comprehension in videos, which is challenging due to complex expression and scene dynamics. Unlike previous methods which solve the problem in multiple stages…
Referring ExpressionReferring Expression ComprehensionVideo GroundingChartREG++: Towards Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referring
Referring expression grounding is a core problem in visual grounding and is widely used as a diagnostic of spatial grounding and reasoning in vision and language models, yet most prior work focuses on natural images. In …
Instance SegmentationReferring ExpressionVisual Grounding