Self-Supervised Learning of Pretext-Invariant Representations
The goal of self-supervised learning from images is to construct image representations that are semantically meaningful via pretext tasks that do not require semantic annotations for a large training set of images. Many pretext tasks lead to representations that are covariant with image transformations. We argue that, instead, semantic representations ought to be invariant under such transformations. Specifically, we develop Pretext-Invariant Representation Learning (PIRL, pronounced as "pearl") that learns invariant representations based on pretext tasks. We use PIRL with a commonly used pretext task that involves solving jigsaw puzzles. We find that PIRL substantially improves the semantic quality of the learned image representations. Our approach sets a new state-of-the-art in self-supervised learning from images on several popular benchmarks for self-supervised learning. Despite being unsupervised, PIRL outperforms supervised pre-training in learning image representations for object detection. Altogether, our results demonstrate the potential of self-supervised learning of image representations with good invariance properties.
Code (7)
Tasks
Contrastive Learningobject-detectionObject DetectionRepresentation LearningSelf-Supervised Image ClassificationSelf-Supervised LearningSemi-Supervised Image ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Downstream Task by Selectively Capturing Complementary Knowledge from Multiple Self-supervisedly Learning Pretexts
Self-supervised learning (SSL), as a newly emerging unsupervised representation learning paradigm, generally follows a two-stage learning pipeline: 1) learning invariant and discriminative representations with auto-annot…
Representation LearningSelf-Supervised LearningSelf-Supervised Learning for Group Equivariant Neural Networks
This paper proposes a method to construct pretext tasks for self-supervised learning on group equivariant neural networks. Group equivariant neural networks are the models whose structure is restricted to commute with th…
Self-Supervised LearningSKID: Self-Supervised Learning for Knee Injury Diagnosis from MRI Data
In medical image analysis, the cost of acquiring high-quality data and their annotation by experts is a barrier in many medical applications. Most of the techniques used are based on supervised learning framework and nee…
Medical DiagnosisMedical Image AnalysisMedical Image ClassificationMulti-Label Classification+2Spectral Aliasing Pretext: A novel task for Self-Supervised fault diagnosis in rotating machinery
Deep learning is a new way for machinery fault diagnosis but requires extensive labeled data, a scarce resource in industrial settings. We propose Spectral Aliasing Pretext (SAP), a self-supervised learning method that p…
Self-Supervised LearningFault DiagnosisText-DIAE: A Self-Supervised Degradation Invariant Autoencoders for Text Recognition and Document Enhancement
In this paper, we propose a Text-Degradation Invariant Auto Encoder (Text-DIAE), a self-supervised model designed to tackle two tasks, text recognition (handwritten or scene-text) and document image enhancement. We start…
Document EnhancementImage EnhancementScene Text Recognition