Learning to Compute Word Embeddings On the Fly
Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare. Learning representations for words in the "long tail" of this distribution requires enormous amounts of data. Representations of rare words trained directly on end tasks are usually poor, requiring us to pre-train embeddings on external data, or treat all rare words as out-of-vocabulary words with a unique representation. We provide a method for predicting embeddings of rare words on the fly from small amounts of auxiliary data with a network trained end-to-end for the downstream task. We show that this improves results against baselines where embeddings are trained on the end task for reading comprehension, recognizing textual entailment and language modeling.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingNatural Language InferenceQuestion AnsweringReading ComprehensionWord EmbeddingsSimilar Papers 제목 키워드 기반
Spanish Biomedical and Clinical Language Embeddings
We computed both Word and Sub-word Embeddings using FastText. For Sub-word embeddings we selected Byte Pair Encoding (BPE) algorithm to represent the sub-words. We evaluated the Biomedical Word Embeddings obtaining bette…
Word EmbeddingsLearning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy
Contextualised word embeddings generated from Neural Language Models (NLMs), such as BERT, represent a word with a vector that considers the semantics of the target word as well its context. On the other hand, static wor…
Word EmbeddingsWord Sense DisambiguationRobust Backed-off Estimation of Out-of-Vocabulary Embeddings
Out-of-vocabulary (oov) words cause serious troubles in solving natural language tasks with a neural network. Existing approaches to this problem resort to using subwords, which are shorter and more ambiguous units than …
Word EmbeddingsWord SimilarityVisualising WordNet Embeddings: some preliminary results
AutoExtend is a method for learning unambiguous vector embeddings for word senses. We visualise these word embeddings with t-SNE, which further compresses the vectors to the x,y plane. We show that the t-SNE co-ordinates…
Semantic SimilaritySemantic Textual SimilarityWord EmbeddingsWord2Sense: Sparse Interpretable Word Embeddings
We present an unsupervised method to generate Word2Sense word embeddings that are interpretable {---} each dimension of the embedding space corresponds to a fine-grained sense, and the non-negative value of the embedding…
Word EmbeddingsWord Similarity