Dynamic Graph Attention for Referring Expression Comprehension
Referring expression comprehension aims to locate the object instance described by a natural language referring expression in an image. This task is compositional and inherently requires visual reasoning on top of the relationships among the objects in the image. Meanwhile, the visual reasoning process is guided by the linguistic structure of the referring expression. However, existing approaches treat the objects in isolation or only explore the first-order relationships between objects without being aligned with the potential complexity of the expression. Thus it is hard for them to adapt to the grounding of complex referring expressions. In this paper, we explore the problem of referring expression comprehension from the perspective of language-driven visual reasoning, and propose a dynamic graph attention network to perform multi-step reasoning by modeling both the relationships among the objects in the image and the linguistic structure of the expression. In particular, we construct a graph for the image with the nodes and edges corresponding to the objects and their relationships respectively, propose a differential analyzer to predict a language-guided visual reasoning process, and perform stepwise reasoning on top of the graph to update the compound object representation at every node. Experimental results demonstrate that the proposed method can not only significantly surpass all existing state-of-the-art algorithms across three common benchmark datasets, but also generate interpretable visual evidences for stepwisely locating the objects referred to in complex language descriptions.
Code (0)
등록된 구현이 없습니다.
Tasks
Graph AttentionReferring ExpressionReferring Expression ComprehensionVisual ReasoningSimilar Papers 제목 키워드 기반
Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networks
The task in referring expression comprehension is to localise the object instance in an image described by a referring expression phrased in natural language. As a language-to-vision matching task, the key to this proble…
Graph AttentionObjectReferring ExpressionReferring Expression ComprehensionCo-Grounding Networks with Semantic Attention for Referring Expression Comprehension in Videos
In this paper, we address the problem of referring expression comprehension in videos, which is challenging due to complex expression and scene dynamics. Unlike previous methods which solve the problem in multiple stages…
Referring ExpressionReferring Expression ComprehensionVideo GroundingThe Solution for the 5th GCAIAC Zero-shot Referring Expression Comprehension Challenge
This report presents a solution for the zero-shot referring expression comprehension task. Visual-language multimodal base models (such as CLIP, SAM) have gained significant attention in recent years as a cornerstone of …
Referring ExpressionReferring Expression ComprehensionMAttNet: Modular Attention Network for Referring Expression Comprehension
In this paper, we address referring expression comprehension: localizing an image region described by a natural language expression. While most recent work treats expressions as a single unit, we propose to decompose the…
Generalized Referring Expression SegmentationReferring ExpressionReferring Expression ComprehensionReferring Expression SegmentationReferring Expression Comprehension: A Survey of Methods and Datasets
Referring expression comprehension (REC) aims to localize a target object in an image described by a referring expression phrased in natural language. Different from the object detection task that queried object labels h…
object-detectionObject DetectionReferring ExpressionReferring Expression Comprehension+1