Issues in evaluating semantic spaces using word analogies
The offset method for solving word analogies has become a standard evaluation tool for vector-space semantic models: it is considered desirable for a space to represent semantic relations as consistent vector offsets. We show that the method's reliance on cosine similarity conflates offset consistency with largely irrelevant neighborhood structure, and propose simple baselines that should be used to improve the utility of the method in vector space evaluation.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Cross-lingual Word Analogies using Linear Transformations between Semantic Spaces
We generalize the word analogy task across languages, to provide a new intrinsic evaluation method for cross-lingual semantic spaces. We experiment with six languages within different language families, including English…
Evaluating vector-space models of analogy
Vector-space representations provide geometric tools for reasoning about the similarity of a set of objects and their relationships. Recent machine learning methods for deriving vector-space embeddings of words (e.g., wo…
Word EmbeddingsTemporal Word Analogies: Identifying Lexical Replacement with Diachronic Word Embeddings
This paper introduces the concept of temporal word analogies: pairs of words which occupy the same semantic space at different points in time. One well-known property of word embeddings is that they are able to effective…
ArticlesDiachronic Word EmbeddingsWord EmbeddingsPortuguese Word Embeddings: Evaluating on Word Analogies and Natural Language Tasks
Word embeddings have been found to provide meaningful representations for words in an efficient way; therefore, they have become common in Natural Language Processing sys- tems. In this paper, we evaluated different word…
POSPOS TaggingSemantic SimilaritySemantic Textual Similarity+2ANALOGICAL -- A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models
Over the past decade, analogies, in the form of word-level analogies, have played a significant role as an intrinsic measure of evaluating the quality of word embedding methods such as word2vec. Modern large language mod…
NegationSentence