Autoencoding Improves Pre-trained Word Embeddings
Prior work investigating the geometry of pre-trained word embeddings have shown that word embeddings to be distributed in a narrow cone and by centering and projecting using principal component vectors one can increase the accuracy of a given set of pre-trained word embeddings. However, theoretically, this post-processing step is equivalent to applying a linear autoencoder to minimise the squared l2 reconstruction error. This result contradicts prior work (Mu and Viswanath, 2018) that proposed to remove the top principal components from pre-trained embeddings. We experimentally verify our theoretical claims and show that retaining the top principal components is indeed useful for improving pre-trained word embeddings, without requiring access to additional linguistic resources or labelled data.
Code (0)
등록된 구현이 없습니다.
Tasks
Word EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Word Meta-Embeddings by Autoencoding
Distributed word embeddings have shown superior performances in numerous Natural Language Processing (NLP) tasks. However, their performances vary significantly across different tasks, implying that the word embeddings l…
Dependency ParsingMachine TranslationPart-Of-Speech TaggingSentiment Analysis+1MoRTy: Unsupervised Learning of Task-specialized Word Embeddings by Autoencoding
Word embeddings have undoubtedly revolutionized NLP. However, pretrained embeddings do not always work for a specific task (or set of tasks), particularly in limited resource setups. We introduce a simple yet effective, …
Word EmbeddingsStructured Neural Topic Models for Reviews
We present Variational Aspect-based Latent Topic Allocation (VALTA), a family of autoencoding topic models that learn aspect-based representations of reviews. VALTA defines a user-item encoder that maps bag-of-words vect…
SentenceTopic ModelsNamed Entity Recognition Only from Word Embeddings
Deep neural network models have helped named entity (NE) recognition achieve amazing performance without handcrafting features. However, existing systems require large amounts of human annotated training data. Efforts ha…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Reinforcement Learning+2Embedding Words as Distributions with a Bayesian Skip-gram Model
We introduce a method for embedding words as probability densities in a low-dimensional space. Rather than assuming that a word embedding is fixed across the entire text collection, as in standard word embedding methods,…