Using Context-to-Vector with Graph Retrofitting to Improve Word Embeddings
Although contextualized embeddings generated from large-scale pre-trained models perform well in many tasks, traditional static embeddings (e.g., Skip-gram, Word2Vec) still play an important role in low-resource and lightweight settings due to their low computational cost, ease of deployment, and stability. In this paper, we aim to improve word embeddings by 1) incorporating more contextual information from existing pre-trained models into the Skip-gram framework, which we call Context-to-Vec; 2) proposing a post-processing retrofitting method for static embeddings independent of training by employing priori synonym knowledge and weighted vector distribution. Through extrinsic and intrinsic tasks, our methods are well proven to outperform the baselines by a large margin.
Code (1)
Tasks
Word EmbeddingsWord SimilaritySimilar Papers 제목 키워드 기반
A Retrofitting Model for Incorporating Semantic Relations into Word Embeddings
We present a novel retrofitting model that can leverage relational knowledge available in a knowledge resource to improve word embeddings. The knowledge is captured in terms of relation inequality constraints that compar…
Lexical EntailmentMetric LearningTripletWord Embeddings+1Embedding Retrofitting: Data Engineering for better RAG
Embedding retrofitting adjusts pre-trained word vectors using knowledge graph constraints to improve domain-specific retrieval. However, the effectiveness of retrofitting depends critically on knowledge graph quality, wh…
Expansional Retrofitting for Word Vector Enrichment
Retrofitting techniques, which inject external resources into word representations, have compensated the weakness of distributed representations in semantic and relational knowledge between words. Implicitly retrofitting…
General Classificationtext-classificationText ClassificationWord SimilaritySpecializing Word Vectors by Spectral Decomposition on Heterogeneously Twisted Graphs
Traditional word vectors, such as word2vec and glove, have a well-known inclination to conflate the semantic similarity with other semantic relations. A retrofitting procedure may be needed to solve this issue. In this w…
Semantic SimilaritySemantic Textual SimilarityExplicit Retrofitting of Distributional Word Vectors
Semantic specialization of distributional word vectors, referred to as retrofitting, is a process of fine-tuning word vectors using external lexical knowledge in order to better embed some semantic relation. Existing ret…
dialog state trackingLexical SimplificationSemantic Textual SimilarityText Simplification+1