Word Embeddings: Stability and Semantic Change
Word embeddings are computed by a class of techniques within natural language processing (NLP), that create continuous vector representations of words in a language from a large text corpus. The stochastic nature of the training process of most embedding techniques can lead to surprisingly strong instability, i.e. subsequently applying the same technique to the same data twice, can produce entirely different results. In this work, we present an experimental study on the instability of the training process of three of the most influential embedding techniques of the last decade: word2vec, GloVe and fastText. Based on the experimental results, we propose a statistical model to describe the instability of embedding techniques and introduce a novel metric to measure the instability of the representation of an individual word. Finally, we propose a method to minimize the instability - by computing a modified average over multiple runs - and apply it to a specific linguistic problem: The detection and quantification of semantic change, i.e. measuring changes in the meaning and usage of words over time.
Code (0)
등록된 구현이 없습니다.
Tasks
Word EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Achieving Semantic Consistency: Contextualized Word Representations for Political Text Analysis
Accurately interpreting words is vital in political science text analysis; some tasks require assuming semantic stability, while others aim to trace semantic shifts. Traditional static embeddings, like Word2Vec effective…
ArticlesInvestigating the Stability of Concrete Nouns in Word Embeddings
We know that word embeddings trained using neural-based methods (such as word2vec SGNS) are sensitive to stability problems and that across two models trained using the exact same set of parameters, the nearest neighbors…
Word EmbeddingsSemantic Change and Semantic Stability: Variation is Key
I survey some recent approaches to studying change in the lexicon, particularly change in meaning across phylogenies. I briefly sketch an evolutionary approach to language change and point out some issues in recent appro…
SurveyWord EmbeddingsUnderstanding the Downstream Instability of Word Embeddings
Many industrial machine learning (ML) systems require frequent retraining to keep up-to-date with constantly changing data. This retraining exacerbates a large challenge facing ML systems today: model training is unstabl…
Word EmbeddingsPredicting Word Embeddings Variability
Neural word embeddings models (such as those built with word2vec) are known to have stability problems: when retraining a model with the exact same hyperparameters, words neighborhoods may change. We propose a method to …
Word Embeddings