On Discriminative Probabilistic Modeling for Self-Supervised Representation Learning
We study the discriminative probabilistic modeling problem on a continuous domain for (multimodal) self-supervised representation learning. To address the challenge of computing the integral in the partition function for each anchor data, we leverage the multiple importance sampling (MIS) technique for robust Monte Carlo integration, which can recover InfoNCE-based contrastive loss as a special case. Within this probabilistic modeling framework, we conduct generalization error analysis to reveal the limitation of current InfoNCE-based contrastive loss for self-supervised representation learning and derive insights for developing better approaches by reducing the error of Monte Carlo integration. To this end, we propose a novel non-parametric method for approximating the sum of conditional densities required by MIS through convex optimization, yielding a new contrastive objective for self-supervised representation learning. Moreover, we design an efficient algorithm for solving the proposed objective. We empirically compare our algorithm to representative baselines on the contrastive image-language pretraining task. Experimental results on the CC3M and CC12M datasets demonstrate the superior overall performance of our algorithm.
Code (0)
등록된 구현이 없습니다.
Tasks
Representation LearningSimilar Papers 제목 키워드 기반
A Probabilistic Model Behind Self-Supervised Learning
In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. A common task is to classify augmentations or different modalities of the data, which share semantic content …
modelRepresentation LearningSelf-Supervised LearningVariational Contrastive Learning for Skeleton-based Action Recognition
In recent years, self-supervised representation learning for skeleton-based action recognition has advanced with the development of contrastive learning methods. However, most of contrastive paradigms are inherently disc…
Self-Supervised LearningRepresentation LearningContrastive LearningAction RecognitionContrastive Separative Coding for Self-supervised Representation Learning
To extract robust deep representations from long sequential modeling of speech data, we propose a self-supervised learning approach, namely Contrastive Separative Coding (CSC). Our key finding is to learn such representa…
Representation LearningSelf-Supervised LearningSpeaker VerificationPrototypical Contrastive Predictive Coding
Transferring representational knowledge of a model to another is a wide-ranging topic in machine learning. Those applications include the distillation of a large supervised or self-supervised teacher model to a smaller s…
Contrastive LearningKnowledge DistillationModel CompressionSelf-Supervised LearningEnhancing Unsupervised Generative Dependency Parser with Contextual Information
Most of the unsupervised dependency parsers are based on probabilistic generative models that learn the joint distribution of the given sentence and its parse. Probabilistic generative models usually explicit decompose t…
Constituency Grammar InductionDependency Grammar InductionSentenceUnsupervised Dependency Parsing