Impart Contextualization to Static Word Embeddings through Semantic Relations
Dense word embedding is the foundational model for the downstream NLP research. It encodes the meanings of words into low dimensional vector spaces. Recent models with the start-of-the-art performances mostly adopt the contextualized word embeddings, which can distinguish the various meanings of the words by their dynamic context. To impart the information of context to the static word embeddings, we formulate 3 semantic relations: interchangeable, opposite and relative relation to find a sub-set of dimensions for interpreting the specific context. The experiment shows that the relations can be mined from fastText embedding.
Code (0)
등록된 구현이 없습니다.
Tasks
Word EmbeddingsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Where exactly does contextualization in a PLM happen?
Pre-trained Language Models (PLMs) have shown to be consistently successful in a plethora of NLP tasks due to their ability to learn contextualized representations of words (Ethayarajh, 2019). BERT (Devlin et al., 2018),…
Language ModelingLanguage ModellingSentenceWord EmbeddingsCitiusNLP at SemEval-2020 Task 3: Comparing Two Approaches for Word Vector Contextualization
This article describes some unsupervised strategies submitted to SemEval 2020 Task 3, a task which consists of considering the effect of context to compute word similarity. More precisely, given two words in context, the…
Word EmbeddingsWord SimilarityPolyLM: Learning about Polysemy through Language Modeling
To avoid the "meaning conflation deficiency" of word embeddings, a number of models have aimed to embed individual word senses. These methods at one time performed well on tasks such as word sense induction (WSI), but th…
Language ModelingLanguage ModellingWord EmbeddingsWord Sense InductionQuantifying the Contextualization of Word Representations with Semantic Class Probing
Pretrained language models have achieved a new state of the art on many NLP tasks, but there are still many open questions about how and why they work so well. We investigate the contextualization of words in BERT. We qu…
Coming to its senses: Lessons learned from Approximating Retrofitted BERT representations for Word Sense information
Retrofitting static vector space word representations using external knowledge bases has yielded substantial improvements in their lexical-semantic capacities but is non-trivial to apply to contextual word embeddings (CW…
Word Embeddings