Learning with Difference Attention for Visually Grounded Self-supervised Representations
Recent works in self-supervised learning have shown impressive results on single-object images, but they struggle to perform well on complex multi-object images as evidenced by their poor visual grounding. To demonstrate this concretely, we propose visual difference attention (VDA) to compute visual attention maps in an unsupervised fashion by comparing an image with its salient-regions-masked-out version. We use VDA to derive attention maps for state-of-the art SSL methods and show they do not highlight all salient regions in an image accurately, suggesting their inability to learn strong representations for downstream tasks like segmentation. Motivated by these limitations, we cast VDA as a differentiable operation and propose a new learning objective, Differentiable Difference Attention (DiDA) loss, which leads to substantial improvements in an SSL model's visually grounding to an image's salient regions.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised LearningVisual GroundingSimilar Papers 제목 키워드 기반
Word Discovery in Visually Grounded, Self-Supervised Speech Models
We present a method for visually-grounded spoken term discovery. After training either a HuBERT or wav2vec2.0 model to associate spoken captions with natural images, we show that powerful word segmentation and clustering…
ClusteringSegmentationVisual GroundingAttending Self-Attention: A Case Study of Visually Grounded Supervision in Vision-and-Language Transformers
The impressive performances of pre-trained visually grounded language models have motivated a growing body of research investigating what has been learned during the pre-training. As a lot of these models are based on Tr…
Language ModelingLanguage ModellingVisual GroundingDoes Structural Attention Improve Compositional Representations in Vision-Language Models?
Although scaling self-supervised approaches has gained widespread success in Vision-Language pre-training, a number of works providing structural knowledge of visually-grounded semantics have recently shown incremental…
Visual ReasoningAttention-Based Keyword Localisation in Speech using Visual Grounding
Visually grounded speech models learn from images paired with spoken captions. By tagging images with soft text labels using a trained visual classifier with a fixed vocabulary, previous work has shown that it is possibl…
Visual GroundingBoosting Visual Instruction Tuning with Self-Supervised Guidance
Multimodal large language models (MLLMs) perform well on many vision-language tasks but often struggle with vision-centric problems that require fine-grained visual reasoning. Recent evidence suggests that this limitatio…
Visual Reasoning