Extremal GloVe: Theoretically Accurate Distributed Word Embedding by Tail Inference
Distributed word embeddings such as Word2Vec and GloVe have been widely adopted in industrial context settings. Major technical applications of GloVe include recommender systems and natural language processing. The fundamental theory behind GloVe relies on the selection of a weighting function in the weighted least squres formulation that computes the powered ratio of word occurrence count and the maximum word count in the corpus. However, the initial formulation of GloVe is not theoretically sound in two aspects, namely the selection of the weighting function and its power exponent is ad-hoc. In this paper, we utilize the theory of extreme value analysis and propose a theoretically accurate version of GloVe. By reformulating the weighted least squares loss function as the expected loss function and accurately choosing the power exponent, we create a theoretically accurate version of GloVe. We demonstrate the competitiveness of our algorithm and show that the initial formulation of GloVe with the suggested optimal parameter can be viewed as a special case of our paradigm.
Code (0)
등록된 구현이 없습니다.
Tasks
Recommendation SystemsWord EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
RankMat : Matrix Factorization with Calibrated Distributed Embedding and Fairness Enhancement
Matrix Factorization is a widely adopted technique in the field of recommender system. Matrix Factorization techniques range from SVD, LDA, pLSA, SVD++, MatRec, Zipf Matrix Factorization and Item2Vec. In recent years, di…
FairnessRecommendation SystemsWord EmbeddingsWhat the Vec? Towards Probabilistically Grounded Embeddings
Word2Vec (W2V) and GloVe are popular, fast and efficient word embedding algorithms. Their embeddings are widely used and perform well on a variety of natural language processing tasks. Moreover, W2V has recently been ado…
Graph EmbeddingWord EmbeddingsDo NLP Models Know Numbers? Probing Numeracy in Embeddings
The ability to understand and work with numbers (numeracy) is critical for many complex reasoning tasks. Currently, most NLP models treat numbers in text in the same way as other tokens---they embed them as distributed v…
Question AnsweringSPINE: SParse Interpretable Neural Embeddings
Prediction without justification has limited utility. Much of the success of neural models can be attributed to their ability to learn rich, dense and expressive representations. While these representations capture the u…
DenoisingWord EmbeddingsPoincaré GloVe: Hyperbolic Word Embeddings
Words are not created equal. In fact, they form an aristocratic graph with a latent hierarchical structure that the next generation of unsupervised learned word embeddings should reveal. In this paper, justified by the n…
Word Embeddings