Neural Sequential Phrase Grounding (SeqGROUND)
We propose an end-to-end approach for phrase grounding in images. Unlike prior methods that typically attempt to ground each phrase independently by building an image-text embedding, our architecture formulates grounding of multiple phrases as a sequential and contextual process. Specifically, we encode region proposals and all phrases into two stacks of LSTM cells, along with so-far grounded phrase-region pairs. These LSTM stacks collectively capture context for grounding of the next phrase. The resulting architecture, which we call SeqGROUND, supports many-to-many matching by allowing an image region to be matched to multiple phrases and vice versa. We show competitive performance on the Flickr30K benchmark dataset and, through ablation studies, validate the efficacy of sequential grounding as well as individual design choices in our model architecture.
Code (0)
등록된 구현이 없습니다.
Tasks
Phrase GroundingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
G3raphGround: Graph-Based Language Grounding
In this paper we present an end-to-end framework for grounding of phrases in images. In contrast to previous works, our model, which we call GraphGround, uses graphs to formulate more complex, non-sequential dependencies…
Graph Neural NetworkDisentangled Motif-aware Graph Learning for Phrase Grounding
In this paper, we propose a novel graph learning framework for phrase grounding in the image. Developing from the sequential to the dense graph model, existing works capture coarse-grained context but fail to distinguish…
DiversityGraph LearningPhrase GroundingExtending Phrase Grounding with Pronouns in Visual Dialogues
Conventional phrase grounding aims to localize noun phrases mentioned in a given caption to their corresponding image regions, which has achieved great success recently. Apparently, sole noun phrase grounding is not enou…
Phrase GroundingImproving Pre-trained Vision-and-Language Embeddings for Phrase Grounding
Phrase grounding aims to map textual phrases to their associated image regions, which can be a prerequisite for multimodal reasoning and can benefit tasks requiring identifying objects based on language. With pre-trained…
Multimodal ReasoningPhrase GroundingA Joint Study of Phrase Grounding and Task Performance in Vision and Language Models
Key to tasks that require reasoning about natural language in visual contexts is grounding words and phrases to image regions. However, observing this grounding in contemporary models is complex, even if it is generally …
Phrase Grounding