Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency
Noise Contrastive Estimation (NCE) is a powerful parameter estimation method for log-linear models, which avoids calculation of the partition function or its derivatives at each training step, a computationally demanding step in many cases. It is closely related to negative sampling methods, now widely used in NLP. This paper considers NCE-based estimation of conditional models. Conditional models are frequently encountered in practice; however there has not been a rigorous theoretical analysis of NCE in this setting, and we will argue there are subtle but important questions when generalizing NCE to the conditional case. In particular, we analyze two variants of NCE for conditional models: one based on a classification objective, the other based on a ranking objective. We show that the ranking-based variant of NCE gives consistent parameter estimates under weaker assumptions than the classification-based method; we analyze the statistical efficiency of the ranking-based and classification-based variants of NCE; finally we describe experiments on synthetic data and language modeling showing the effectiveness and trade-offs of both methods.
Code (0)
등록된 구현이 없습니다.
Tasks
ClassificationGeneral ClassificationLanguage ModelingLanguage Modellingparameter estimationQuestion AnsweringSimilar Papers 제목 키워드 기반
Conditional Negative Sampling for Contrastive Learning of Visual Representations
Recent methods for learning unsupervised visual representations, dubbed contrastive learning, optimize the noise-contrastive estimation (NCE) bound on mutual information between two views of an image. NCE uses randomly s…
Contrastive LearningInstance SegmentationKeypoint DetectionLinear evaluation+3On the connection between Noise-Contrastive Estimation and Contrastive Divergence
Noise-contrastive estimation (NCE) is a popular method for estimating unnormalised probabilistic models, such as energy-based models, which are effective for modelling complex data distributions. Unlike classical maximum…
Mutual Information Estimation as a Difference of Entropies for Unsupervised Representation Learning
Contrastive loss has been successfully exploited in the latest visual unsupervised representation learning methods. Contrastive loss is based on a lower-bound estimation of mutual information where its known limitations …
Mutual Information EstimationRepresentation LearningNotes on Noise Contrastive Estimation and Negative Sampling
Estimating the parameters of probabilistic models of language such as maxent models and probabilistic neural models is computationally difficult since it involves evaluating partition functions by summing over an entire …
Binary Classificationparameter estimationDo More Negative Samples Necessarily Hurt in Contrastive Learning?
Recent investigations in noise contrastive estimation suggest, both empirically as well as theoretically, that while having more "negative samples" in the contrastive loss improves downstream classification performance i…
Contrastive Learning