Learning Word Meta-Embeddings by Autoencoding
Distributed word embeddings have shown superior performances in numerous Natural Language Processing (NLP) tasks. However, their performances vary significantly across different tasks, implying that the word embeddings learnt by those methods capture complementary aspects of lexical semantics. Therefore, we believe that it is important to combine the existing word embeddings to produce more accurate and complete meta-embeddings of words. We model the meta-embedding learning problem as an autoencoding problem, where we would like to learn a meta-embedding space that can accurately reconstruct all source embeddings simultaneously. Thereby, the meta-embedding space is enforced to capture complementary information in different source embeddings via a coherent common embedding space. We propose three flavours of autoencoded meta-embeddings motivated by different requirements that must be satisfied by a meta-embedding. Our experimental results on a series of benchmark evaluations show that the proposed autoencoded meta-embeddings outperform the existing state-of-the-art meta-embeddings in multiple tasks.
Code (1)
Tasks
Dependency ParsingMachine TranslationPart-Of-Speech TaggingSentiment AnalysisWord EmbeddingsSimilar Papers 제목 키워드 기반
Autoencoding Improves Pre-trained Word Embeddings
Prior work investigating the geometry of pre-trained word embeddings have shown that word embeddings to be distributed in a narrow cone and by centering and projecting using principal component vectors one can increase t…
Word EmbeddingsMoRTy: Unsupervised Learning of Task-specialized Word Embeddings by Autoencoding
Word embeddings have undoubtedly revolutionized NLP. However, pretrained embeddings do not always work for a specific task (or set of tasks), particularly in limited resource setups. We introduce a simple yet effective, …
Word EmbeddingsThink Globally, Embed Locally --- Locally Linear Meta-embedding of Words
Distributed word embeddings have shown superior performances in numerous Natural Language Processing (NLP) tasks. However, their performances vary significantly across different tasks, implying that the word embeddings l…
General ClassificationRelation ClassificationSemantic SimilaritySemantic Textual Similarity+3Embedding Words as Distributions with a Bayesian Skip-gram Model
We introduce a method for embedding words as probability densities in a low-dimensional space. Rather than assuming that a word embedding is fixed across the entire text collection, as in standard word embedding methods,…
Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings
Given multiple source word embeddings learnt using diverse algorithms and lexical resources, meta word embedding learning methods attempt to learn more accurate and wide-coverage word embeddings. Prior work on meta-embed…
Word Embeddings