Visual Grounding with Transformers
In this paper, we propose a transformer based approach for visual grounding. Unlike previous proposal-and-rank frameworks that rely heavily on pretrained object detectors or proposal-free frameworks that upgrade an off-the-shelf one-stage detector by fusing textual embeddings, our approach is built on top of a transformer encoder-decoder and is independent of any pretrained detectors or word embedding models. Termed VGTR -- Visual Grounding with TRansformers, our approach is designed to learn semantic-discriminative visual features under the guidance of the textual description without harming their location ability. This information flow enables our VGTR to have a strong capability in capturing context-level semantics of both vision and language modalities, rendering us to aggregate accurate visual clues implied by the description to locate the interested object instance. Experiments show that our method outperforms state-of-the-art proposal-free approaches by a considerable margin on five benchmarks while maintaining fast inference speed.
Code (1)
Tasks
DecoderSimilar Papers 제목 키워드 기반
Weakly Supervised Grounding for VQA in Vision-Language Transformers
Transformers for visual-language representation learning have been getting a lot of interest and shown tremendous performance on visual question answering (VQA) and grounding. But most systems that show good performance …
Question AnsweringRepresentation LearningVisual Question AnsweringVisual Question Answering (VQA)Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling
This study explores replacing Transformers in Visual Language Models (VLMs) with Mamba, a recent structured state space model (SSM) that demonstrates promising performance in sequence modeling. We test models up to 3B pa…
Language ModelingLanguage ModellingMambaQuestion Answering+4Human-centric Spatio-Temporal Video Grounding With Visual Transformers
In this work, we introduce a novel task - Humancentric Spatio-Temporal Video Grounding (HC-STVG). Unlike the existing referring expression tasks in images or videos, by focusing on humans, HC-STVG aims to localize a spat…
Referring ExpressionSentenceSpatio-Temporal Video GroundingTemporal Localization+1Attending Self-Attention: A Case Study of Visually Grounded Supervision in Vision-and-Language Transformers
The impressive performances of pre-trained visually grounded language models have motivated a growing body of research investigating what has been learned during the pre-training. As a lot of these models are based on Tr…
Language ModelingLanguage ModellingVisual GroundingTransVG: End-to-End Visual Grounding with Transformers
In this paper, we present a neat yet effective transformer-based framework for visual grounding, namely TransVG, to address the task of grounding a language query to the corresponding region onto an image. The state-of-t…
Referring Expression ComprehensionVisual Grounding