G3raphGround: Graph-Based Language Grounding
In this paper we present an end-to-end framework for grounding of phrases in images. In contrast to previous works, our model, which we call GraphGround, uses graphs to formulate more complex, non-sequential dependencies among proposal image regions and phrases. We capture intra-modal dependencies using a separate graph neural network for each modality (visual and lingual), and then use conditional message-passing in another graph neural network to fuse their outputs and capture cross-modal relationships. This final representation results in grounding decisions. The framework supports many-to-many matching and is able to ground single phrase to multiple image regions and vice versa. We validate our design choices through a series of ablation studies and illustrate state-of-the-art performance on Flickr30k and ReferIt Game benchmark datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
Graph Neural NetworkMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
SceneGraphGrounder: Zero-Shot 3D Visual Grounding via Structured Scene Graph Matching
Zero-shot 3D visual grounding requires localizing objects in unstructured environments from free-form natural language. Recent vision-language model (VLM) approaches achieve promising results but rely on view-dependent r…
Spatial ReasoningVisual GroundingGraph MatchingLearning Cross-modal Context Graph for Visual Grounding
Visual grounding is a ubiquitous building block in many vision-language tasks and yet remains challenging due to large variations in visual and linguistic features of grounding entities, strong context effect and the res…
Graph MatchingGraph Neural NetworkLanguage ModellingNatural Language Visual Grounding+2Learning Cross-modal Context Graph for Visual Grounding
Visual grounding is a ubiquitous building block in many vision-language tasks and yet remains challenging due to large variations in visual and linguistic features of grounding entities, strong context effect and the res…
Graph MatchingGraph Neural NetworkVisual GroundingDeconfounded Visual Grounding
We focus on the confounding bias between language and location in the visual grounding pipeline, where we find that the bias is the major visual reasoning bottleneck. For example, the grounding process is usually a trivi…
Referring ExpressionVisual GroundingVisual ReasoningDifferentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placement
We present a new method, PARsing And visual GrOuNding (ParaGon), for grounding natural language in object placement tasks. Natural language generally describes objects and spatial relations with compositionality and ambi…
Graph Neural NetworkObjectRelational ReasoningVisual Grounding