A Closer Look at Self-Supervised Lightweight Vision Transformers
Self-supervised learning on large-scale Vision Transformers (ViTs) as pre-training methods has achieved promising downstream performance. Yet, how much these pre-training paradigms promote lightweight ViTs' performance is considerably less studied. In this work, we develop and benchmark several self-supervised pre-training methods on image classification tasks and some downstream dense prediction tasks. We surprisingly find that if proper pre-training is adopted, even vanilla lightweight ViTs show comparable performance to previous SOTA networks with delicate architecture design. It breaks the recently popular conception that vanilla ViTs are not suitable for vision tasks in lightweight regimes. We also point out some defects of such pre-training, e.g., failing to benefit from large-scale pre-training data and showing inferior performance on data-insufficient downstream tasks. Furthermore, we analyze and clearly show the effect of such pre-training by analyzing the properties of the layer representation and attention maps for related models. Finally, based on the above analyses, a distillation strategy during pre-training is developed, which leads to further downstream performance improvement for MAE-based pre-training. Code is available at https://github.com/wangsr126/mae-lite.
Code (2)
Tasks
Contrastive Learningimage-classificationImage ClassificationSelf-Supervised LearningSimilar Papers 제목 키워드 기반
A Closer Look at Invariances in Self-supervised Pre-training for 3D Vision
Self-supervised pre-training for 3D vision has drawn increasing research interest in recent years. In order to learn informative representations, a lot of previous works exploit invariances of 3D features, e.g., perspect…
Contrastive Learningobject-detectionObject DetectionA Closer Look at Few-Shot Out-of-Distribution Intent Detection
We consider few-shot out-of-distribution (OOD) intent detection, a practical and important problem for the development of task-oriented dialogue systems. Despite its importance, this problem is seldom studied in the lite…
Intent DetectionTask-Oriented Dialogue SystemsConditional Contrastive Learning for Improving Fairness in Self-Supervised Learning
Contrastive self-supervised learning (SSL) learns an embedding space that maps similar data pairs closer and dissimilar data pairs farther apart. Despite its success, one issue has been overlooked: the fairness aspect of…
AttributeContrastive LearningFairnessRepresentation Learning+1SelfKG: Self-Supervised Entity Alignment in Knowledge Graphs
Entity alignment, aiming to identify equivalent entities across different knowledge graphs (KGs), is a fundamental problem for constructing Web-scale KGs. Over the course of its development, the label supervision has bee…
Entity AlignmentKnowledge GraphsSelf-Supervised LearningA Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification
Self-supervised learning (SSL) is a machine learning approach where the data itself provides supervision, eliminating the need for external labels. The model is forced to learn about the data structure or context by solv…
BenchmarkingFew-Shot Learningimage-classificationImage Classification+1