Unsupervised Transformer Pre-Training for Images: Self-Distillation, Mean Teachers, and Random Crops
Recent advances in self-supervised learning (SSL) have made it possible to learn general-purpose visual features that capture both the high-level semantics and the fine-grained spatial structure of images. Most notably, the recent DINOv2 has established a new state of the art by surpassing weakly supervised methods (WSL) like OpenCLIP on most benchmarks. In this survey, we examine the core ideas behind its approach, multi-crop view augmentation and self-distillation with a mean teacher, and trace their development in previous work. We then compare the performance of DINO and DINOv2 with other SSL and WSL methods across various downstream tasks, and highlight some remarkable emergent properties of their learned features with transformer backbones. We conclude by briefly discussing DINOv2's limitations, its impact, and future research directions.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised LearningSimilar Papers 제목 키워드 기반
ProS: Facial Omni-Representation Learning via Prototype-based Self-Distillation
This paper presents a novel approach, called Prototype-based Self-Distillation (ProS), for unsupervised face representation learning. The existing supervised methods heavily rely on a large amount of annotated training f…
AttributeRepresentation LearningCo-distilled attention guided masked image modeling with noisy teacher for self-supervised learning on medical images
Masked image modeling (MIM) is a highly effective self-supervised learning (SSL) approach to extract useful feature representations from unannotated data. Predominantly used random masking methods make SSL less effective…
Lung Nodule ClassificationSelf-Supervised LearningTumor SegmentationDearKD: Data-Efficient Early Knowledge Distillation for Vision Transformers
Transformers are successfully applied to computer vision due to their powerful modeling capacity with self-attention. However, the excellent performance of transformers heavily depends on enormous training images. Thus, …
Knowledge DistillationUncovering the Inner Workings of STEGO for Safe Unsupervised Semantic Segmentation
Self-supervised pre-training strategies have recently shown impressive results for training general-purpose feature extraction backbones in computer vision. In combination with the Vision Transformer architecture, the DI…
Dimensionality ReductionSemantic SegmentationUnsupervised Semantic SegmentationSelf-Supervised Keypoint Detection with Distilled Depth Keypoint Representation
Existing unsupervised keypoint detection methods apply artificial deformations to images such as masking a significant portion of images and using reconstruction of original image as a learning objective to detect keypoi…
Keypoint DetectionKnowledge Distillation