An experimental analysis of Noise-Contrastive Estimation: the noise distribution matters
Noise Contrastive Estimation (NCE) is a learning procedure that is regularly used to train neural language models, since it avoids the computational bottleneck caused by the output softmax. In this paper, we attempt to explain some of the weaknesses of this objective function, and to draw directions for further developments. Experiments on a small task show the issues raised by an unigram noise distribution, and that a context dependent noise distribution, such as the bigram distribution, can solve these issues and provide stable and data-efficient learning.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingMachine TranslationSpeech RecognitionSimilar Papers 제목 키워드 기반
Self-Adapting Noise-Contrastive Estimation for Energy-Based Models
Training energy-based models (EBMs) with noise-contrastive estimation (NCE) is theoretically feasible but practically challenging. Effective learning requires the noise distribution to be approximately similar to the tar…
Data Augmentation of Contrastive Learning is Estimating Positive-incentive Noise
Inspired by the idea of Positive-incentive Noise (Pi-Noise or $\pi$-Noise) that aims at learning the reliable noise beneficial to tasks, we scientifically investigate the connection between contrastive learning and $\pi$…
Contrastive LearningData AugmentationLearn Beneficial Noise as Graph Augmentation
Although graph contrastive learning (GCL) has been widely investigated, it is still a challenge to generate effective and stable graph augmentations. Existing methods often apply heuristic augmentation like random edge d…
Contrastive LearningData AugmentationFully Variational Noise-Contrastive Estimation
By using the underlying theory of proper scoring rules, we design a family of noise-contrastive estimation (NCE) methods that are tractable for latent variable models. Both terms in the underlying NCE loss, the one using…
Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learning
Contrastive representation learning has proven to be an effective self-supervised learning method. Most successful approaches are based on Noise Contrastive Estimation (NCE) and use different views of an instance as posi…
Contrastive LearningLinear evaluationRepresentation LearningSelf-Supervised Image Classification+3