On the connection between Noise-Contrastive Estimation and Contrastive Divergence
Noise-contrastive estimation (NCE) is a popular method for estimating unnormalised probabilistic models, such as energy-based models, which are effective for modelling complex data distributions. Unlike classical maximum likelihood (ML) estimation that relies on importance sampling (resulting in ML-IS) or MCMC (resulting in contrastive divergence, CD), NCE uses a proxy criterion to avoid the need for evaluating an often intractable normalisation constant. Despite apparent conceptual differences, we show that two NCE criteria, ranking NCE (RNCE) and conditional NCE (CNCE), can be viewed as ML estimation methods. Specifically, RNCE is equivalent to ML estimation combined with conditional importance sampling, and both RNCE and CNCE are special cases of CD. These findings bridge the gap between the two method classes and allow us to apply techniques from the ML-IS and CD literature to NCE, offering several advantageous extensions.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Data Augmentation of Contrastive Learning is Estimating Positive-incentive Noise
Inspired by the idea of Positive-incentive Noise (Pi-Noise or $\pi$-Noise) that aims at learning the reliable noise beneficial to tasks, we scientifically investigate the connection between contrastive learning and $\pi$…
Contrastive LearningData AugmentationYour contrastive learning problem is secretly a distribution alignment problem
Despite the success of contrastive learning (CL) in vision and language, its theoretical foundations and mechanisms for building representations remain poorly understood. In this work, we build connections between noise …
Contrastive LearningSelf-Supervised LearningFrom $t$-SNE to UMAP with contrastive learning
Neighbor embedding methods $t$-SNE and UMAP are the de facto standard for visualizing high-dimensional datasets. Motivated from entirely different viewpoints, their loss functions appear to be unrelated. In practice, the…
Contrastive LearningRepresentation LearningSimilarity Contrastive Estimation for Image and Video Soft Contrastive Self-Supervised Learning
Contrastive representation learning has proven to be an effective self-supervised learning method for images and videos. Most successful approaches are based on Noise Contrastive Estimation (NCE) and use different views …
Contrastive LearningLinear evaluationRepresentation LearningSelf-Supervised Action Recognition+4Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learning
Contrastive representation learning has proven to be an effective self-supervised learning method. Most successful approaches are based on Noise Contrastive Estimation (NCE) and use different views of an instance as posi…
Contrastive LearningLinear evaluationRepresentation LearningSelf-Supervised Image Classification+3