Prototypical Contrastive Learning of Unsupervised Representations
This paper presents Prototypical Contrastive Learning (PCL), an unsupervised representation learning method that addresses the fundamental limitations of instance-wise contrastive learning. PCL not only learns low-level features for the task of instance discrimination, but more importantly, it implicitly encodes semantic structures of the data into the learned embedding space. Specifically, we introduce prototypes as latent variables to help find the maximum-likelihood estimation of the network parameters in an Expectation-Maximization framework. We iteratively perform E-step as finding the distribution of prototypes via clustering and M-step as optimizing the network via contrastive learning. We propose ProtoNCE loss, a generalized version of the InfoNCE loss for contrastive learning, which encourages representations to be closer to their assigned prototypes. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks with substantial improvement in low-resource transfer learning. Code and pretrained models are available at https://github.com/salesforce/PCL.
Code (2)
Tasks
ClusteringContrastive LearningRepresentation LearningSelf-Supervised Image ClassificationSemi-Supervised Image ClassificationTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Contrastive Prototypical Network with Wasserstein Confidence Penalty
Unsupervised few-shot learning aims to learn the inductive bias from unlabeled dataset for solving the novel few-shot tasks. The existing unsupervised few-shot learning models and the contrastive learning models follow a…
Contrastive LearningFew-Shot LearningInductive BiasUnsupervised Few-Shot Image Classification+1Unsupervised Feature Learning by Autoencoder and Prototypical Contrastive Learning for Hyperspectral Classification
Unsupervised learning methods for feature extraction are becoming more and more popular. We combine the popular contrastive learning method (prototypical contrastive learning) and the classic representation learning meth…
Contrastive LearningGeneral ClassificationRepresentation LearningRethinking Prototypical Contrastive Learning through Alignment, Uniformity and Correlation
Contrastive self-supervised learning (CSL) with a prototypical regularization has been introduced in learning meaningful representations for downstream tasks that require strong semantic information. However, to optimize…
Contrastive LearningDiversitySelf-Supervised LearningBayesian Robust Graph Contrastive Learning
Graph Neural Networks (GNNs) have been widely used to learn node representations and with outstanding performance on various tasks such as node classification. However, noise, which inevitably exists in real-world graph …
Contrastive LearningNode ClassificationPrototypical Contrastive Predictive Coding
Transferring representational knowledge of a model to another is a wide-ranging topic in machine learning. Those applications include the distillation of a large supervised or self-supervised teacher model to a smaller s…
Contrastive LearningKnowledge DistillationModel CompressionSelf-Supervised Learning