Retrofitting Contextualized Word Embeddings with Paraphrases
Contextualized word embedding models, such as ELMo, generate meaningful representations of words and their context. These models have been shown to have a great impact on downstream applications. However, in many cases, the contextualized embedding of a word changes drastically when the context is paraphrased. As a result, the downstream model is not robust to paraphrasing and other linguistic variations. To enhance the stability of contextualized word embedding models, we propose an approach to retrofitting contextualized embedding models with paraphrase contexts. Our method learns an orthogonal transformation on the input space, which seeks to minimize the variance of word representations on paraphrased contexts. Experiments show that the retrofitted model significantly outperforms the original ELMo on various sentence classification and language inference tasks.
Code (0)
등록된 구현이 없습니다.
Tasks
SentenceSentence ClassificationWord EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Using Paraphrases to Study Properties of Contextual Embeddings
We use paraphrases as a unique source of data to analyze contextualized embeddings, with a particular focus on BERT. Because paraphrases naturally encode consistent word and phrase semantics, they provide a unique lens f…
Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings
Although contextualized embeddings generated from large-scale pre-trained models perform well in many tasks, traditional static embeddings (e.g., Skip-gram, Word2Vec) still play an important role in low-resource and ligh…
Word EmbeddingsWord SimilarityIntegrating Lexical Knowledge in Word Embeddings using Sprinkling and Retrofitting
Neural network based word embeddings, such as Word2Vec and GloVe, are purely data driven in that they capture the distributional information about words from the training corpus. Past works have attempted to improve thes…
Word EmbeddingsWord Sense Induction using Knowledge Embeddings
Word Embeddings are able to capture lexico-semantic information but remain flawed in their inability to assign unique representations to different senses of a polysemous words. They also fail to include information from …
Word EmbeddingsWord Sense InductionExploring Fine-Tuned Embeddings that Model Intensifiers for Emotion Analysis
Adjective phrases like "a little bit surprised", "completely shocked", or "not stunned at all" are not handled properly by currently published state-of-the-art emotion classification and intensity prediction systems whic…
Emotion ClassificationEmotion RecognitionGeneral ClassificationWord Embeddings