ScanFormer: Referring Expression Comprehension by Iteratively Scanning
Referring Expression Comprehension (REC) aims to localize the target objects specified by free-form natural language descriptions in images. While state-of-the-art methods achieve impressive performance, they perform a dense perception of images, which incorporates redundant visual regions unrelated to linguistic queries, leading to additional computational overhead. This inspires us to explore a question: can we eliminate linguistic-irrelevant redundant visual regions to improve the efficiency of the model? Existing relevant methods primarily focus on fundamental visual tasks, with limited exploration in vision-language fields. To address this, we propose a coarse-to-fine iterative perception framework, called ScanFormer. It can iteratively exploit the image scale pyramid to extract linguistic-relevant visual patches from top to bottom. In each iteration, irrelevant patches are discarded by our designed informativeness prediction. Furthermore, we propose a patch selection strategy for discarded patches to accelerate inference. Experiments on widely used datasets, namely RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame, verify the effectiveness of our method, which can strike a balance between accuracy and efficiency.
Code (0)
등록된 구현이 없습니다.
Tasks
InformativenessReferring ExpressionReferring Expression ComprehensionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Comprehension-guided referring expressions
We consider generation and comprehension of natural language referring expression for objects in an image. Unlike generic "image captioning" which lacks natural standard evaluation criteria, quality of a referring expres…
Referring ExpressionReferring expression generationThe Solution for the 5th GCAIAC Zero-shot Referring Expression Comprehension Challenge
This report presents a solution for the zero-shot referring expression comprehension task. Visual-language multimodal base models (such as CLIP, SAM) have gained significant attention in recent years as a cornerstone of …
Referring ExpressionReferring Expression ComprehensionA Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehension
Referring expression comprehension aims to localize the object instance described by a natural language expression. Current referring expression methods have achieved good performance. However, none of them is able to ac…
Referring ExpressionReferring Expression ComprehensionNeighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networks
The task in referring expression comprehension is to localise the object instance in an image described by a referring expression phrased in natural language. As a language-to-vision matching task, the key to this proble…
Graph AttentionObjectReferring ExpressionReferring Expression ComprehensionA Joint Speaker-Listener-Reinforcer Model for Referring Expressions
Referring expressions are natural language constructions used to identify particular objects within a scene. In this paper, we propose a unified framework for the tasks of referring expression comprehension and generatio…
Referring ExpressionReferring Expression Comprehension