Self-supervised Pretraining of Visual Features in the Wild
Recently, self-supervised learning methods like MoCo, SimCLR, BYOL and SwAV have reduced the gap with supervised methods. These results have been achieved in a control environment, that is the highly curated ImageNet dataset. However, the premise of self-supervised learning is that it can learn from any random image and from any unbounded dataset. In this work, we explore if self-supervision lives to its expectation by training large models on random, uncurated images with no supervision. Our final SElf-supERvised (SEER) model, a RegNetY with 1.3B parameters trained on 1B random images with 512 GPUs achieves 84.2% top-1 accuracy, surpassing the best self-supervised pretrained model by 1% and confirming that self-supervised learning works in a real world setting. Interestingly, we also observe that self-supervised models are good few-shot learners achieving 77.9% top-1 with access to only 10% of ImageNet. Code: https://github.com/facebookresearch/vissl
Code (1)
Tasks
Image ClassificationSelf-Supervised Image ClassificationSelf-Supervised LearningSemi-Supervised Image ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Self-supervised learning improves robustness of deep learning lung tumor segmentation to CT imaging differences
Self-supervised learning (SSL) is an approach to extract useful feature representations from unlabeled data, and enable fine-tuning on downstream tasks with limited labeled examples. Self-pretraining is a SSL approach th…
Computed Tomography (CT)Medical Image AnalysisSelf-Supervised LearningTumor SegmentationLeveraging In-the-Wild Data for Effective Self-Supervised Pretraining in Speaker Recognition
Current speaker recognition systems primarily rely on supervised approaches, constrained by the scale of labeled datasets. To boost the system performance, researchers leverage large pretrained models such as WavLM to tr…
Speaker RecognitionEfficient Visual Pretraining with Contrastive Detection
Self-supervised pretraining has been shown to yield powerful representations for transfer learning. These performance gains come at a large computational cost however, with state-of-the-art methods requiring an order of …
Semantic SegmentationTransfer LearningWildlife Target Re-Identification Using Self-supervised Learning in Non-Urban Settings
Wildlife re-identification aims to match individuals of the same species across different observations. Current state-of-the-art (SOTA) models rely on class labels to train supervised models for individual classification…
Self-Supervised LearningTransfer LearningConnect Later: Improving Fine-tuning for Robustness with Targeted Augmentations
Models trained on a labeled source domain (e.g., labeled images from wildlife camera traps) often generalize poorly when deployed on an out-of-distribution (OOD) target domain (e.g., images from new camera trap locations…
Contrastive LearningDomain AdaptationTime SeriesTime Series Classification