The Word Analogy Testing Caveat
There are some important problems in the evaluation of word embeddings using standard word analogy tests. In particular, in virtue of the assumptions made by systems generating the embeddings, these remain tests over randomness. We show that even supposing there were such word analogy regularities that should be detected in the word embeddings obtained via unsupervised means, standard word analogy test implementation practices provide distorted or contrived results. We raise concerns regarding the use of Principal Component Analysis to 2 or 3 dimensions as a provision of visual evidence for the existence of word analogy relations in embeddings. Finally, we propose some solutions to these problems.
Code (0)
등록된 구현이 없습니다.
Tasks
Semantic Textual SimilarityTransfer LearningWord EmbeddingsSimilar Papers 제목 키워드 기반
Word Embeddings from Large-Scale Greek Web Content
Word embeddings are undoubtedly very useful components in many NLP tasks. In this paper, we present word embeddings and other linguistic resources trained on the largest to date digital Greek language corpus. We also pre…
Word EmbeddingsCaveats of Measuring Semantic Change of Cognates and Borrowings using Multilingual Word Embeddings
Cognates and borrowings carry different aspects of etymological evolution. In this work, we study semantic change of such items using multilingual word embeddings, both static and contextualised. We underline caveats ide…
Multilingual Word EmbeddingsWord EmbeddingsSemantic Regularities in Document Representations
Recent work exhibited that distributed word representations are good at capturing linguistic regularities in language. This allows vector-oriented reasoning based on simple linear algebra between words. Since many differ…
Traversal-Free Word Vector Evaluation in Analogy Space
In this paper, we propose an alternative evaluating metric for word analogy questions (A to B is as C to D) in word vector evaluation. Different from the traditional method which predicts the fourth word by the given thr…
KaWAT: A Word Analogy Task Dataset for Indonesian
We introduced KaWAT (Kata Word Analogy Task), a new word analogy task dataset for Indonesian. We evaluated on it several existing pretrained Indonesian word embeddings and embeddings trained on Indonesian online news cor…
Word Embeddings