Affinity-based Attention in Self-supervised Transformers Predicts Dynamics of Object Grouping in Humans
The spreading of attention has been proposed as a mechanism for how humans group features to segment objects. However, such a mechanism has not yet been implemented and tested in naturalistic images. Here, we leverage the feature maps from self-supervised vision Transformers and propose a model of human object-based attention spreading and segmentation. Attention spreads within an object through the feature affinity signal between different patches of the image. We also collected behavioral data on people grouping objects in natural images by judging whether two dots are on the same object or on two different objects. We found that our models of affinity spread that were built on feature maps from the self-supervised Transformers showed significant improvement over baseline and CNN based models on predicting reaction time patterns of humans, despite not being trained on the task or with any other object labels. Our work provides new benchmarks for evaluating models of visual representation learning including Transformers.
Code (1)
Tasks
ObjectRepresentation LearningSimilar Papers 제목 키워드 기반
Learning Affinity from Attention: End-to-End Weakly-Supervised Semantic Segmentation with Transformers
Weakly-supervised semantic segmentation (WSSS) with image-level labels is an important and challenging task. Due to the high training efficiency, end-to-end solutions for WSSS have received increasing attention from the …
Semantic SegmentationWeakly supervised Semantic SegmentationWeakly-Supervised Semantic SegmentationThe Origin of Self-Attention: Pairwise Affinity Matrices in Feature Selection and the Emergence of Self-Attention
The self-attention mechanism, now central to deep learning architectures such as Transformers, is a modern instance of a more general computational principle: learning and using pairwise affinity matrices to control how …
Graph LearningAll-pairs Consistency Learning for Weakly Supervised Semantic Segmentation
In this work, we propose a new transformer-based regularization to better localize objects for Weakly supervised semantic segmentation (WSSS). In image-level WSSS, Class Activation Map (CAM) is adopted to generate object…
AllObject LocalizationSemantic SegmentationWeakly supervised Semantic Segmentation+1CAST: Clustering Self-Attention using Surrogate Tokens for Efficient Transformers
The Transformer architecture has shown to be a powerful tool for a wide range of tasks. It is based on the self-attention mechanism, which is an inherently computationally expensive operation with quadratic computational…
ClusteringA Self-Distillation Embedded Supervised Affinity Attention Model for Few-Shot Segmentation
Few-shot segmentation focuses on the generalization of models to segment unseen object with limited annotated samples. However, existing approaches still face two main challenges. First, huge feature distinction between …
Few-Shot Semantic SegmentationSegmentationSemantic Segmentation