Towards Incremental Learning of Word Embeddings Using Context Informativeness
In this paper, we investigate the task of learning word embeddings from very sparse data in an incremental, cognitively-plausible way. We focus on the notion of {`}informativeness{'}, that is, the idea that some content is more valuable to the learning process than other. We further highlight the challenges of online learning and argue that previous systems fall short of implementing incrementality. Concretely, we incorporate informativeness in a previously proposed model of nonce learning, using it for context selection and learning rate modulation. We test our system on the task of learning new words from definitions, as well as on the task of learning new words from potentially uninformative contexts. We demonstrate that informativeness is crucial to obtaining state-of-the-art performance in a truly incremental setup.
Code (1)
Tasks
Incremental LearningInformativenessLearning Word EmbeddingsWord EmbeddingsSimilar Papers 제목 키워드 기반
An Attention-Based Model for Predicting Contextual Informativeness and Curriculum Learning Applications
Both humans and machines learn the meaning of unknown words through contextual information in a sentence, but not all contexts are equally helpful for learning. We introduce an effective method for capturing the level of…
InformativenessSentenceExtending Text Informativeness Measures to Passage Interestingness Evaluation (Language Model vs. Word Embedding)
Standard informativeness measures used to evaluate Automatic Text Summarization mostly rely on n-gram overlapping between the automatic summary and the reference summaries. These measures differ from the metric they use …
Information RetrievalInformativenessLanguage ModelingLanguage Modelling+3Exploring sentence informativeness
This study is a preliminary exploration of the concept of informativeness -how much information a sentence gives about a word it contains- and its potential benefits to building quality word representations from scarce d…
InformativenessSentenceWord EmbeddingsDecoding, Fast and Slow: A Case Study on Balancing Trade-Offs in Incremental, Character-level Pragmatic Reasoning
Recent work has adopted models of pragmatic reasoning for the generation of informative language in, e.g., image captioning. We propose a simple but highly effective relaxation of fully rational decoding, based on an exi…
Image CaptioningInformativenessLanguage ModelingLanguage Modelling+1Incremental Sense Weight Training for the Interpretation of Contextualized Word Embeddings
We present a novel online algorithm that learns the essence of each dimension in word embeddings by minimizing the within-group distance of contextualized embedding groups. Three state-of-the-art neural-based language mo…
Word EmbeddingsWord Sense Disambiguation