Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts
Learning high-quality embeddings for rare words is a hard problem because of sparse context information. Mimicking (Pinter et al., 2017) has been proposed as a solution: given embeddings learned by a standard algorithm, a model is first trained to reproduce embeddings of frequent words from their surface form and then used to compute embeddings for rare words. In this paper, we introduce attentive mimicking: the mimicking model is given access not only to a word's surface form, but also to all available contexts and learns to attend to the most informative and reliable contexts for computing an embedding. In an evaluation on four tasks, we show that attentive mimicking outperforms previous work for both rare and medium-frequency words. Thus, compared to previous work, attentive mimicking improves embeddings for a much larger part of the vocabulary, including the medium-frequency range.
Code (1)
Tasks
FormWord EmbeddingsSimilar Papers 제목 키워드 기반
Rare Words: A Major Problem for Contextualized Embeddings And How to Fix it by Attentive Mimicking
Pretraining deep neural network architectures with a language modeling objective has brought large improvements for many natural language processing tasks. Exemplified by BERT, a recently proposed such architecture, we d…
Language ModelingLanguage ModellingWord Sense Induction with Attentive Context Clustering
In this paper, we present ACCWSI (Attentive Context Clustering WSI), a method for Word Sense Induction, suitable for languages with limited resources. Pretrained on a small corpus and given an ambiguous word (query word)…
ClusteringWord Sense InductionImproving Disfluency Detection by Self-Training a Self-Attentive Model
Self-attentive neural syntactic parsers using contextualized word embeddings (e.g. ELMo or BERT) currently produce state-of-the-art results in joint parsing and disfluency detection in speech transcripts. Since the conte…
Word EmbeddingsQuasi Bidirectional Encoder Representations from Transformers for Word Sense Disambiguation
While contextualized embeddings have produced performance breakthroughs in many Natural Language Processing (NLP) tasks, Word Sense Disambiguation (WSD) has not benefited from them yet. In this paper, we introduce QBERT,…
Word Sense DisambiguationAttentive Neural Network for Named Entity Recognition in Vietnamese
We propose an attentive neural network for the task of named entity recognition in Vietnamese. The proposed attentive neural model makes use of character-based language models and word embeddings to encode words as vecto…
Decodernamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition In Vietnamese+2