Investigating Self-Supervised Methods for Label-Efficient Learning
Vision transformers combined with self-supervised learning have enabled the development of models which scale across large datasets for several downstream tasks like classification, segmentation and detection. The low-shot learning capability of these models, across several low-shot downstream tasks, has been largely under explored. We perform a system level study of different self supervised pretext tasks, namely contrastive learning, clustering, and masked image modelling for their low-shot capabilities by comparing the pretrained models. In addition we also study the effects of collapse avoidance methods, namely centring, ME-MAX, sinkhorn, on these downstream tasks. Based on our detailed analysis, we introduce a framework involving both mask image modelling and clustering as pretext tasks, which performs better across all low-shot downstream tasks, including multi-class classification, multi-label classification and semantic segmentation. Furthermore, when testing the model on full scale datasets, we show performance gains in multi-class classification, multi-label classification and semantic segmentation.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationClusteringContrastive LearningMulti-class ClassificationMulti-Label ClassificationMUlTI-LABEL-ClASSIFICATIONSegmentationSelf-Supervised LearningSemantic SegmentationSimilar Papers 제목 키워드 기반
Investigating a Baseline Of Self Supervised Learning Towards Reducing Labeling Costs For Image Classification
Data labeling in supervised learning is considered an expensive and infeasible tool in some conditions. The self-supervised learning method is proposed to tackle the learning effectiveness with fewer labeled data, howeve…
image-classificationImage ClassificationSelf-Supervised LearningThe Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning
Self-supervised methods have become crucial for advancing deep learning by leveraging data itself to reduce the need for expensive annotations. However, the question of how to conduct self-supervised offline reinforcemen…
Offline RLreinforcement-learningReinforcement Learning (RL)An Experimental Comparison Of Multi-view Self-supervised Methods For Music Tagging
Self-supervised learning has emerged as a powerful way to pre-train generalizable machine learning models on large amounts of unlabeled data. It is particularly compelling in the music domain, where obtaining labeled dat…
Contrastive LearningMusic TaggingSelf-Supervised LearningContrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
The success of learning with noisy labels (LNL) methods relies heavily on the success of a warm-up stage where standard supervised training is performed using the full (noisy) training set. In this paper, we identify a "…
Image ClassificationLearning with noisy labelsMemorizationInvestigating Semi-Supervised Learning Algorithms in Text Datasets
Using large training datasets enhances the generalization capabilities of neural networks. Semi-supervised learning (SSL) is useful when there are few labeled data and a lot of unlabeled data. SSL methods that use data a…
Data Augmentation