Towards Lower Bounds on Number of Dimensions for Word Embeddings
Word embeddings are a relatively new addition to the modern NLP researcher{'}s toolkit. However, unlike other tools, word embeddings are used in a black box manner. There are very few studies regarding various hyperparameters. One such hyperparameter is the dimension of word embeddings. They are rather decided based on a rule of thumb: in the range 50 to 300. In this paper, we show that the dimension should instead be chosen based on corpus statistics. More specifically, we show that the number of pairwise equidistant words of the corpus vocabulary (as defined by some distance/similarity metric) gives a lower bound on the the number of dimensions , and going below this bound results in degradation of quality of learned word embeddings. Through our evaluations on standard word embedding evaluation tasks, we show that for dimensions higher than or equal to the bound, we get better results as compared to the ones below it.
Code (0)
등록된 구현이 없습니다.
Tasks
Named Entity Recognition (NER)Part-Of-Speech TaggingSarcasm DetectionSentence ClassificationSentiment AnalysisWord EmbeddingsSimilar Papers 제목 키워드 기반
Compact Geometric Representations of Hierarchies
Computing geometric representations of data is a cornerstone of modern machine learning, typically achieved by training dual encoders which map queries and documents into a shared embedding space. Recent work of You et a…
Embeddings and labeling schemes for A*
A* is a classic and popular method for graphs search and path finding. It assumes the existence of a heuristic function $h(u,t)$ that estimates the shortest distance from any input node $u$ to the destination $t$. Tradit…
A Gapped Scale-Sensitive Dimension and Lower Bounds for Offset Rademacher Complexity
We study gapped scale-sensitive dimensions of a function class in both sequential and non-sequential settings. We demonstrate that covering numbers for any uniformly bounded class are controlled above by these gapped dim…
Binary Encoded Word Mover’s Distance
Word Mover’s Distance is a textual distance metric which calculates the minimum transport cost between two sets of word embeddings. This metric achieves impressive results on semantic similarity tasks, but is slow and di…
Semantic SimilaritySemantic Textual SimilarityWord EmbeddingsRepresentation Tradeoffs for Hyperbolic Embeddings
Hyperbolic embeddings offer excellent quality with few dimensions when embedding hierarchical data structures like synonym or type hierarchies. Given a tree, we give a combinatorial construction that embeds the tree in h…