The Role of Context Types and Dimensionality in Learning Word Embeddings
We provide the first extensive evaluation of how using different types of context to learn skip-gram word embeddings affects performance on a wide range of intrinsic and extrinsic NLP tasks. Our results suggest that while intrinsic tasks tend to exhibit a clear preference to particular types of contexts and higher dimensionality, more careful tuning is required for finding the optimal settings for most of the extrinsic tasks that we considered. Furthermore, for these extrinsic tasks, we find that once the benefit from increasing the embedding dimensionality is mostly exhausted, simple concatenation of word embeddings, learned with different context types, can yield further performance gains. As an additional contribution, we propose a new variant of the skip-gram model that learns word embeddings from weighted contexts of substitute words.
Code (0)
등록된 구현이 없습니다.
Tasks
Learning Word EmbeddingsWord EmbeddingsSimilar Papers 제목 키워드 기반
Effect of dimensionality change on the bias of word embeddings
Word embedding methods (WEMs) are extensively used for representing text data. The dimensionality of these embeddings varies across various tasks and implementations. The effect of dimensionality change on the accuracy o…
Word EmbeddingsLearning Contextual Hierarchical Structure of Medical Concepts with Poincairé Embeddings to Clarify Phenotypes
Biomedical association studies are increasingly done using clinical concepts, and in particular diagnostic codes from clinical data repositories as phenotypes. Clinical concepts can be represented in a meaningful, vector…
BIG-bench Machine LearningDiagnosticEmbeddings in Natural Language Processing
Embeddings have been one of the most important topics of interest in NLP for the past decade. Representing knowledge through a low-dimensional vector which is easily integrable in modern machine learning models has playe…
High-Level SynthesisWord EmbeddingsSimple and Effective Dimensionality Reduction for Word Embeddings
Word embeddings have become the basic building blocks for several natural language processing and information retrieval tasks. Pre-trained word embeddings are used in several downstream applications as well as for constr…
Dimensionality ReductionInformation RetrievalRetrievalWord Embeddings+1Leveraging Three Types of Embeddings from Masked Language Models in Idiom Token Classification
Many linguistic expressions have idiomatic and literal interpretations, and the automatic distinction of these two interpretations has been studied for decades. Recent research has shown that contextualized word embeddin…
Classificationtoken-classificationToken ClassificationWord Embeddings