Bringing Order to Neural Word Embeddings with Embeddings Augmented by Random Permutations (EARP)
Word order is clearly a vital part of human language, but it has been used comparatively lightly in distributional vector models. This paper presents a new method for incorporating word order information into word vector embedding models by combining the benefits of permutation-based order encoding with the more recent method of skip-gram with negative sampling. The new method introduced here is called Embeddings Augmented by Random Permutations (EARP). It operates by applying permutations to the coordinates of context vector representations during the process of training. Results show an 8{\%} improvement in accuracy on the challenging Bigger Analogy Test Set, and smaller but consistent improvements on other analogy reference sets. These findings demonstrate the importance of order-based information in analogical retrieval tasks, and the utility of random permutations as a means to augment neural embeddings.
Code (0)
등록된 구현이 없습니다.
Tasks
RetrievalWord EmbeddingsSimilar Papers 제목 키워드 기반
Quantifying Context Overlap for Training Word Embeddings
Most models for learning word embeddings are trained based on the context information of words, more precisely first order co-occurrence relations. In this paper, a metric is designed to estimate second order co-occurren…
Dimensionality ReductionLanguage ModelingLanguage ModellingLearning Word Embeddings+4Sub-Word Similarity based Search for Embeddings: Inducing Rare-Word Embeddings for Word Similarity Tasks and Language Modelling
Training good word embeddings requires large amounts of data. Out-of-vocabulary words will still be encountered at test-time, leaving these words without embeddings. To overcome this lack of embeddings for rare words, ex…
Language ModelingLanguage ModellingMorphological AnalysisWord Embeddings+1On Learning Word Embeddings From Linguistically Augmented Text Corpora
Word embedding is a technique in Natural Language Processing (NLP) to map words into vector space representations. Since it has boosted the performance of many NLP downstream tasks, the task of learning word embeddings h…
Learning Word EmbeddingsWord EmbeddingsWord SimilarityNeural-based Noise Filtering from Word Embeddings
Word embeddings have been demonstrated to benefit NLP tasks impressively. Yet, there is room for improvement in the vector representations, because current word embeddings typically contain unnecessary information, i.e.,…
DenoisingWord EmbeddingsEfficient, Compositional, Order-sensitive n-gram Embeddings
We propose ECO: a new way to generate embeddings for phrases that is Efficient, Compositional, and Order-sensitive. Our method creates decompositional embeddings for words offline and combines them to create new embeddin…
Word Embeddings