Problems With Evaluation of Word Embeddings Using Word Similarity Tasks
Lacking standardized extrinsic evaluation methods for vector representations of words, the NLP community has relied heavily on word similarity tasks as a proxy for intrinsic evaluation of word vectors. Word similarity evaluation, which correlates the distance between vectors and human judgments of semantic similarity is attractive, because it is computationally inexpensive and fast. In this paper we present several problems associated with the evaluation of word vectors on word similarity datasets, and summarize existing solutions. Our study suggests that the use of word similarity tasks for evaluation of word vectors is not sustainable and calls for further research on evaluation methods.
Code (1)
Tasks
Semantic SimilaritySemantic Textual SimilarityWord EmbeddingsWord SimilaritySimilar Papers 제목 키워드 기반
Word-Alignment-Based Segment-Level Machine Translation Evaluation using Word Embeddings
One of the most important problems in machine translation (MT) evaluation is to evaluate the similarity between translation hypotheses with different surface forms from the reference, especially at the segment level. We …
Machine TranslationTranslationWord AlignmentWord EmbeddingsJust Rank: Rethinking Evaluation with Word and Sentence Similarities
Word and sentence embeddings are useful feature representations in natural language processing. However, intrinsic evaluation for embeddings lags far behind, and there has been no significant update since the past decade…
BenchmarkingSemantic SimilaritySemantic Textual SimilaritySentence+4HHU at SemEval-2017 Task 2: Fast Hash-Based Embeddings for Semantic Word Similarity Assessment
This paper describes the HHU system that participated in Task 2 of SemEval 2017, Multilingual and Cross-lingual Semantic Word Similarity. We introduce our unsupervised embedding learning technique and describe how it was…
Learning Word EmbeddingsSemantic Textual SimilarityTask 2Word Embeddings+1Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words
Cosine similarity of contextual embeddings is used in many NLP tasks (e.g., QA, IR, MT) and metrics (e.g., BERTScore). Here, we uncover systematic ways in which word similarities estimated by cosine over BERT embeddings …
Meta-Embedding as Auxiliary Task Regularization
Word embeddings have been shown to benefit from ensambling several word embedding sources, often carried out using straightforward mathematical operations over the set of word vectors. More recently, self-supervised lear…
Self-Supervised LearningSentenceSentence ClassificationWord Embeddings+1