Patch-level Representation Learning for Self-supervised Vision Transformers
Recent self-supervised learning (SSL) methods have shown impressive results in learning visual representations from unlabeled images. This paper aims to improve their performance further by utilizing the architectural advantages of the underlying neural network, as the current state-of-the-art visual pretext tasks for SSL do not enjoy the benefit, i.e., they are architecture-agnostic. In particular, we focus on Vision Transformers (ViTs), which have gained much attention recently as a better architectural choice, often outperforming convolutional networks for various visual tasks. The unique characteristic of ViT is that it takes a sequence of disjoint patches from an image and processes patch-level representations internally. Inspired by this, we design a simple yet effective visual pretext task, coined SelfPatch, for learning better patch-level representations. To be specific, we enforce invariance against each patch and its neighbors, i.e., each patch treats similar neighboring patches as positive samples. Consequently, training ViTs with SelfPatch learns more semantically meaningful relations among patches (without using human-annotated labels), which can be beneficial, in particular, to downstream tasks of a dense prediction type. Despite its simplicity, we demonstrate that it can significantly improve the performance of existing SSL methods for various visual tasks, including object detection and semantic segmentation. Specifically, SelfPatch significantly improves the recent self-supervised ViT, DINO, by achieving +1.3 AP on COCO object detection, +1.2 AP on COCO instance segmentation, and +2.9 mIoU on ADE20K semantic segmentation.
Code (1)
Tasks
Instance Segmentationobject-detectionObject DetectionRepresentation LearningSegmentationSelf-Supervised LearningSemantic SegmentationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PASS: Patch-Aware Self-Supervision for Vision Transformer
Recent self-supervised representation learning methods have shown impressive results in learning visual representations from unlabeled images. This paper aims to improve their performance further by utilizing the archite…
object-detectionObject DetectionRepresentation LearningSelf-Supervised Learning+1Self-Supervised Pyramid Representation Learning for Multi-Label Visual Analysis and Beyond
While self-supervised learning has been shown to benefit a number of vision tasks, existing techniques mainly focus on image-level manipulation, which may not generalize well to downstream tasks at patch or pixel levels.…
Instance SegmentationMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONobject-detection+4Patch-Wise Self-Supervised Visual Representation Learning: A Fine-Grained Approach
Self-supervised visual representation learning traditionally focuses on image-level instance discrimination. Our study introduces an innovative, fine-grained dimension by integrating patch-level discrimination into these…
Copy Detectionimage-classificationImage ClassificationImage Retrieval+3Masked Siamese Networks for Label-Efficient Learning
We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our approach matches the representation of an image view containing randomly masked patches to the repres…
image-classificationImage ClassificationSelf-Supervised Image ClassificationSelf-Supervised Learning+1Pattern Integration and Enhancement Vision Transformer for Self-Supervised Learning in Remote Sensing
Recent self-supervised learning (SSL) methods have demonstrated impressive results in learning visual representations from unlabeled remote sensing images. However, most remote sensing images predominantly consist of sce…
Change DetectionLand Cover Classificationobject-detectionObject Detection+2