An Empirical Study of the Downstream Reliability of Pre-Trained Word Embeddings
While pre-trained word embeddings have been shown to improve the performance of downstream tasks, many questions remain regarding their reliability: Do the same pre-trained word embeddings result in the best performance with slight changes to the training data? Do the same pre-trained embeddings perform well with multiple neural network architectures? Do imputation strategies for unknown words impact reliability? In this paper, we introduce two new metrics to understand the downstream reliability of word embeddings. We find that downstream reliability of word embeddings depends on multiple factors, including, the evaluation metric, the handling of out-of-vocabulary words, and whether the embeddings are fine-tuned.
Code (0)
등록된 구현이 없습니다.
Tasks
ImputationWord EmbeddingsSimilar Papers 제목 키워드 기반
k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations
Since the inception of crowdsourcing, aggregation has been a common strategy for dealing with unreliable data. Aggregate ratings are more reliable than individual ones. However, many NLP datasets that rely on aggregate r…
Word SimilarityGating Mechanisms for Combining Character and Word-level Word Representations: An Empirical Study
In this paper we study how different ways of combining character and word-level representations affect the quality of both final word and sentence representations. We provide strong empirical evidence that modeling chara…
Semantic SimilaritySemantic Textual SimilaritySentenceWord SimilarityHow do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese
This paper investigates the effect of tokenizers on the downstream performance of pretrained language models (PLMs) in scriptio continua languages where no explicit spaces exist between words, using Japanese as a case st…
Can CLIP Count Stars? An Empirical Study on Quantity Bias in CLIP
CLIP has demonstrated great versatility in adapting to various downstream tasks, such as image editing and generation, visual question answering, and video understanding. However, CLIP-based applications often suffer fro…
Image GenerationQuestion AnsweringVideo UnderstandingVisual Question AnsweringWord2Vec: Optimal Hyper-Parameters and Their Impact on NLP Downstream Tasks
Word2Vec is a prominent model for natural language processing (NLP) tasks. Similar inspiration is found in distributed embeddings for new state-of-the-art (SotA) deep neural networks. However, wrong combination of hyper-…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NER+1