PASS: Patch-Aware Self-Supervision for Vision Transformer
Recent self-supervised representation learning methods have shown impressive results in learning visual representations from unlabeled images. This paper aims to improve their performance further by utilizing the architectural advantages of the underlying neural network, as the current state-of-the-art visual pretext tasks for self-supervised learning do not enjoy the benefit, i.e., they are architecture-agnostic. In particular, we focus on Vision Transformers (ViTs), which have gained much attention recently as a better architectural choice, often outperforming convolutional networks for various visual tasks. The unique characteristic of ViT is that it takes a sequence of disjoint patches from an input image and processes patch-level representations internally. Inspired by this, we design a simple yet effective visual pretext task, coined Patch-Aware Self-Supervision (PASS), for learning better patch-level representations. To be specific, we enforce invariance against each patch and its neighbors, i.e., each patch treats similar neighboring patches as positive samples. Consequently, training ViTs with PASS produces more semantically meaningful attention maps patch-wisely in an unsupervised manner, which can be beneficial, in particular, to downstream tasks of a dense prediction type. Despite the simplicity of our scheme, we demonstrate that it can significantly improve the performance of existing self-supervised learning methods for various visual tasks, including object detection and semantic segmentation.
Code (0)
등록된 구현이 없습니다.
Tasks
object-detectionObject DetectionRepresentation LearningSelf-Supervised LearningSemantic SegmentationSimilar Papers 제목 키워드 기반
Self-Promoted Supervision for Few-Shot Transformer
The few-shot learning ability of vision transformers (ViTs) is rarely investigated though heavily desired. In this work, we empirically find that with the same few-shot learning frameworks, \eg~Meta-Baseline, replacing t…
Data AugmentationFew-Shot LearningInductive BiasSelf-Supervised Representation Learning on Document Images
This work analyses the impact of self-supervised pre-training on document images in the context of document image classification. While previous approaches explore the effect of self-supervision on natural images, we sho…
Classificationdocument-image-classificationDocument Image ClassificationGeneral Classification+3Mix-and-Match Tuning for Self-Supervised Semantic Segmentation
Deep convolutional networks for semantic image segmentation typically require large-scale labeled data, e.g. ImageNet and MS COCO, for network pre-training. To reduce annotation efforts, self-supervised semantic segmenta…
ColorizationImage ColorizationImage SegmentationSegmentation+2MammoDINO: Anatomically Aware Self-Supervision for Mammographic Images
Self-supervised learning (SSL) has transformed vision encoder training in general domains but remains underutilized in medical imaging due to limited data and domain specific biases. We present MammoDINO, a novel SSL fra…
Self-Supervised LearningContrastive LearningData AugmentationAdvancing Melanoma Diagnosis with Self-Supervised Neural Networks: Evaluating the Effectiveness of Different Techniques
We investigate the potential of self-supervision in improving the accuracy of deep learning models trained to classify melanoma patches. Various self-supervision techniques such as rotation prediction, missing patch pred…
Contrastive LearningMelanoma Diagnosis