Self-Supervised Learning by Estimating Twin Class Distributions
We present TWIST, a simple and theoretically explainable self-supervised representation learning method by classifying large-scale unlabeled datasets in an end-to-end way. We employ a siamese network terminated by a softmax operation to produce twin class distributions of two augmented images. Without supervision, we enforce the class distributions of different augmentations to be consistent. However, simply minimizing the divergence between augmentations will cause collapsed solutions, i.e., outputting the same class probability distribution for all images. In this case, no information about the input image is left. To solve this problem, we propose to maximize the mutual information between the input and the class predictions. Specifically, we minimize the entropy of the distribution for each sample to make the class prediction for each sample assertive and maximize the entropy of the mean distribution to make the predictions of different samples diverse. In this way, TWIST can naturally avoid the collapsed solutions without specific designs such as asymmetric network, stop-gradient operation, or momentum encoder. As a result, TWIST outperforms state-of-the-art methods on a wide range of tasks. Especially, TWIST performs surprisingly well on semi-supervised learning, achieving 61.2% top-1 accuracy with 1% ImageNet labels using a ResNet-50 as backbone, surpassing previous best results by an absolute improvement of 6.2%. Codes and pre-trained models are given on: https://github.com/bytedance/TWIST
Code (2)
Tasks
Fine-Grained Image ClassificationImage ClassificationRepresentation LearningSelf-Supervised Image ClassificationSelf-Supervised LearningSemi-Supervised Image ClassificationTransfer LearningUnsupervised Image ClassificationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Audio Barlow Twins: Self-Supervised Audio Representation Learning
The Barlow Twins self-supervised learning objective requires neither negative samples or asymmetric learning updates, achieving results on a par with the current state-of-the-art within Computer Vision. As such, we prese…
Environmental Sound ClassificationEvent DetectionRepresentation LearningSelf-Supervised LearningBarlow Twins: Self-Supervised Learning via Redundancy Reduction
Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. A successful approach to SSL is to learn embeddings which are invariant to distortions of the input s…
General ClassificationImage ClassificationObject DetectionSelf-Supervised Image Classification+2Infinite Width Limits of Self Supervised Neural Networks
The NTK is a widely used tool in the theoretical analysis of deep learning, allowing us to look at supervised deep neural networks through the lenses of kernel regression. Recently, several works have investigated kernel…
Self-Supervised LearningSelf-supervised Knowledge Distillation for Few-shot Learning
Real-world contains an overwhelmingly large number of object classes, learning all of which at once is infeasible. Few shot learning is a promising learning paradigm due to its ability to learn out of order distributions…
Few-Shot Image ClassificationFew-Shot LearningKnowledge DistillationMeta-Learning+1A Histopathology Study Comparing Contrastive Semi-Supervised and Fully Supervised Learning
Data labeling is often the most challenging task when developing computational pathology models. Pathologist participation is necessary to generate accurate labels, and the limitations on pathologist time and demand for …
Active LearningSelf-Supervised LearningWeakly-supervised Learning