Investigating the Frequency Distortion of Word Embeddings and Its Impact on Bias Metrics
Recent research has shown that static word embeddings can encode word frequency information. However, little has been studied about this phenomenon and its effects on downstream tasks. In the present work, we systematically study the association between frequency and semantic similarity in several static word embeddings. We find that Skip-gram, GloVe and FastText embeddings tend to produce higher semantic similarity between high-frequency words than between other frequency combinations. We show that the association between frequency and similarity also appears when words are randomly shuffled. This proves that the patterns found are not due to real semantic associations present in the texts, but are an artifact produced by the word embeddings. Finally, we provide an example of how word frequency can strongly impact the measurement of gender bias with embedding-based metrics. In particular, we carry out a controlled experiment that shows that biases can even change sign or reverse their order by manipulating word frequencies.
Code (1)
Tasks
Semantic SimilaritySemantic Textual SimilarityWord EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Frequency-based Distortions in Contextualized Word Embeddings
How does word frequency in pre-training data affect the behavior of similarity metrics in contextualized BERT embeddings? Are there systematic ways in which some word relationships are exaggerated or understated? In this…
Semantic SimilaritySemantic Textual SimilarityWord EmbeddingsFrequency-aware Dimension Selection for Static Word Embedding by Mixed Product Distance
Static word embedding is still useful, particularly for context-unavailable tasks, because in the case of no context available, pre-trained language models often perform worse than static word embeddings. Although dimens…
Word EmbeddingsInvestigating the Stability of Concrete Nouns in Word Embeddings
We know that word embeddings trained using neural-based methods (such as word2vec SGNS) are sensitive to stability problems and that across two models trained using the exact same set of parameters, the nearest neighbors…
Word EmbeddingsMeasuring Similarity by Linguistic Features rather than Frequency
In the use and creation of current Deep Learning Models the only number that is used for the overall computation is the frequency value associated with the current word form in the corpus, which is used to substitute it.…
Word EmbeddingsEvaluating Bias In Dutch Word Embeddings
Recent research in Natural Language Processing has revealed that word embeddings can encode social biases present in the training data which can affect minorities in real world applications. This paper explores the gende…
ClusteringSentenceSentence EmbeddingsWord Embeddings