Improving Word Representations: A Sub-sampled Unigram Distribution for Negative Sampling
Word2Vec is the most popular model for word representation and has been widely investigated in literature. However, its noise distribution for negative sampling is decided by empirical trials and the optimality has always been ignored. We suggest that the distribution is a sub-optimal choice, and propose to use a sub-sampled unigram distribution for better negative sampling. Our contributions include: (1) proposing the concept of semantics quantification and deriving a suitable sub-sampling rate for the proposed distribution adaptive to different training corpora; (2) demonstrating the advantages of our approach in both negative sampling and noise contrastive estimation by extensive evaluation tasks; and (3) proposing a semantics weighted model for the MSR sentence completion task, resulting in considerable improvements. Our work not only improves the quality of word vectors but also benefits current understanding of Word2Vec.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceSentence CompletionSimilar Papers 제목 키워드 기반
Modeling the Unigram Distribution
The unigram distribution is the non-contextual probability of finding a specific word form in a corpus. While of central importance to the study of language, it is commonly approximated by each word's sample frequency in…
FormNorm of Word Embedding Encodes Information Gain
Distributed representations of words encode lexical semantic information, but what type of information is encoded and how? Focusing on the skip-gram with negative-sampling method, we found that the squared norm of static…
InformativenessKeyword ExtractionProper NounWord EmbeddingsSemantic Composition and Decomposition: From Recognition to Generation
Semantic composition is the task of understanding the meaning of text by composing the meanings of the individual words in the text. Semantic decomposition is the task of understanding the meaning of an individual word b…
Semantic CompositionAuthor-Specific Linguistic Patterns Unveiled: A Deep Learning Study on Word Class Distributions
Deep learning methods have been increasingly applied to computational linguistics to uncover patterns in text data. This study investigates author-specific word class distributions using part-of-speech (POS) tagging and …
Author ProfilingPart-Of-Speech TaggingPOSPOS Tagging+1Benchmarks Are Not That Out of Distribution: Word Overlap Predicts Performance
Understanding what constitutes high-quality pre-training data remains a central question in language model training. In this work, we investigate whether benchmark performance is primarily driven by the degree of statist…