Quantifying Context Overlap for Training Word Embeddings
Most models for learning word embeddings are trained based on the context information of words, more precisely first order co-occurrence relations. In this paper, a metric is designed to estimate second order co-occurrence relations based on context overlap. The estimated values are further used as the augmented data to enhance the learning of word embeddings by joint training with existing neural word embedding models. Experimental results show that better word vectors can be obtained for word similarity tasks and some downstream NLP tasks by the enhanced approach.
Code (0)
등록된 구현이 없습니다.
Tasks
Dimensionality ReductionLanguage ModelingLanguage ModellingLearning Word EmbeddingsRepresentation LearningSemantic CompositionWord EmbeddingsWord SimilaritySimilar Papers 제목 키워드 기반
On the Downstream Performance of Compressed Word Embeddings
Compressing word embeddings is important for deploying NLP models in memory-constrained settings. However, understanding what makes compressed embeddings perform well on downstream tasks is challenging---existing measure…
Generalization BoundsQuantizationWord EmbeddingsFair Embedding Engine: A Library for Analyzing and Mitigating Gender Bias in Word Embeddings
Non-contextual word embedding models have been shown to inherit human-like stereotypical biases of gender, race and religion from the training corpora. To counter this issue, a large body of research has emerged which ai…
Word EmbeddingsAn exploration of the encoding of grammatical gender in word embeddings
The vector representation of words, known as word embeddings, has opened a new research approach in linguistic studies. These representations can capture different types of information about words. The grammatical gender…
ArticlesWord EmbeddingsQuantifying Compositionality of Classic and State-of-the-Art Embeddings
For language models to generalize correctly to novel expressions, it is critical that they exploit access compositional meanings when this is justified. Even if we don't know what a "pelp" is, we can use our knowledge of…
Knowledge GraphsQuantifying the Contextualization of Word Representations with Semantic Class Probing
Pretrained language models have achieved a new state of the art on many NLP tasks, but there are still many open questions about how and why they work so well. We investigate the contextualization of words in BERT. We qu…