Context Vectors are Reflections of Word Vectors in Half the Dimensions
This paper takes a step towards theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec. We start from basic probabilistic assumptions on the nature of word vectors, context vectors, and text generation. These assumptions are well supported either empirically or theoretically by the existing literature. Next, we show that under these assumptions the widely-used word-word PMI matrix is approximately a random symmetric Gaussian ensemble. This, in turn, implies that context vectors are reflections of word vectors in approximately half the dimensions. As a direct application of our result, we suggest a theoretically grounded way of tying weights in the SGNS model.
Code (0)
등록된 구현이 없습니다.
Tasks
Text GenerationWord EmbeddingsSimilar Papers 제목 키워드 기반
Causally Denoise Word Embeddings Using Half-Sibling Regression
Distributional representations of words, also known as word vectors, have become crucial for modern natural language processing tasks due to their wide applications. Recently, a growing body of word vector postprocessing…
Causal InferenceregressionSentiment AnalysisWord EmbeddingsWord and Document Embedding with vMF-Mixture Priors on Context Word Vectors
Word embedding models typically learn two types of vectors: target word vectors and context word vectors. These vectors are normally learned such that they are predictive of some word co-occurrence statistic, but they ar…
Document EmbeddingLearning Word Embeddings without Context Vectors
Most word embedding algorithms such as word2vec or fastText construct two sort of vectors: for words and for contexts. Naive use of vectors of only one sort leads to poor results. We suggest using indefinite inner produc…
Learning Word EmbeddingsWord EmbeddingsWord SimilarityA Rule-based/BPSO Approach to Produce Low-dimensional Semantic Basis Vectors Set
We intend to generate low-dimensional explicit distributional semantic vectors. In explicit semantic vectors, each dimension corresponds to a word, so word vectors are interpretable. In this research, we propose a new ap…
Word SimilarityModelling General Properties of Nouns by Selectively Averaging Contextualised Embeddings
While the success of pre-trained language models has largely eliminated the need for high-quality static word vectors in many NLP applications, such vectors continue to play an important role in tasks where words need to…
Knowledge Base Completion