Self-supervised Learning: Generative or Contrastive
Deep supervised learning has achieved great success in the last decade. However, its deficiencies of dependence on manual labels and vulnerability to attacks have driven people to explore a better solution. As an alternative, self-supervised learning attracts many researchers for its soaring performance on representation learning in the last several years. Self-supervised representation learning leverages input data itself as supervision and benefits almost all types of downstream tasks. In this survey, we take a look into new self-supervised learning methods for representation in computer vision, natural language processing, and graph learning. We comprehensively review the existing empirical methods and summarize them into three main categories according to their objectives: generative, contrastive, and generative-contrastive (adversarial). We further investigate related theoretical analysis work to provide deeper thoughts on how self-supervised learning works. Finally, we briefly discuss open problems and future directions for self-supervised learning. An outline slide for the survey is provided.
Code (0)
등록된 구현이 없습니다.
Tasks
Graph LearningRepresentation LearningSelf-Supervised LearningSurveySimilar Papers 제목 키워드 기반
Generative or Contrastive? Phrase Reconstruction for Better Sentence Representation Learning
Though offering amazing contextualized token-level representations, current pre-trained language models actually take less attention on acquiring sentence-level representation during its self-supervised pre-training. If …
Contrastive LearningRepresentation LearningRetrievalSelf-Supervised Learning+3Self-Supervised Learning for Time Series: Contrastive or Generative?
Self-supervised learning (SSL) has recently emerged as a powerful approach to learning representations from large-scale unlabeled data, showing promising results in time series analysis. The self-supervised representatio…
Model OptimizationRepresentation LearningSelf-Supervised LearningTime Series+1A Framework for Generative and Contrastive Learning of Audio Representations
In this paper, we present a framework for contrastive learning for audio representations, in a self supervised frame work without access to any ground truth labels. The core idea in self supervised contrastive learning i…
Contrastive LearningJoint Generative-Contrastive Representation Learning for Anomalous Sound Detection
In this paper, we propose a joint generative and contrastive representation learning method (GeCo) for anomalous sound detection (ASD). GeCo exploits a Predictive AutoEncoder (PAE) equipped with self-attention as a gener…
Contrastive LearningRepresentation LearningThe Power of Contrast for Feature Learning: A Theoretical Analysis
Contrastive learning has achieved state-of-the-art performance in various self-supervised learning tasks and even outperforms its supervised counterpart. Despite its empirical success, theoretical understanding of the su…
Contrastive LearningSelf-Supervised LearningTransfer Learning