SemGloVe: Semantic Co-occurrences for GloVe from BERT
GloVe learns word embeddings by leveraging statistical information from word co-occurrence matrices. However, word pairs in the matrices are extracted from a predefined local context window, which might lead to limited word pairs and potentially semantic irrelevant word pairs. In this paper, we propose SemGloVe, which distills semantic co-occurrences from BERT into static GloVe word embeddings. Particularly, we propose two models to extract co-occurrence statistics based on either the masked language model or the multi-head attention weights of BERT. Our methods can extract word pairs without limiting by the local window assumption and can define the co-occurrence weights by directly considering the semantic distance between word pairs. Experiments on several word similarity datasets and four external tasks show that SemGloVe can outperform GloVe.
Code (3)
Tasks
Language ModelingLanguage ModellingWord EmbeddingsWord SimilarityMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Word-Node2Vec: Improving Word Embedding with Document-Level Non-Local Word Co-occurrences
A standard word embedding algorithm, such as word2vec and glove, makes a strong assumption that words are likely to be semantically related only if they co-occur locally within a window of fixed size. However, this stron…
ViCo: Word Embeddings from Visual Co-occurrences
We propose to learn word embeddings from visual co-occurrences. Two words co-occur visually if both words apply to the same image or image region. Specifically, we extract four types of visual co-occurrences between obje…
AttributeClusteringWord EmbeddingsSST-BERT at SemEval-2020 Task 1: Semantic Shift Tracing by Clustering in BERT-based Embedding Spaces
Lexical semantic change detection (also known as semantic shift tracing) is a task of identifying words that have changed their meaning over time. Unsupervised semantic shift tracing, focal point of SemEval2020, is parti…
Change DetectionClusteringThe aftermath of compounds: Investigating Compounds and their Semantic Representations
This study investigates how well computational embeddings align with human semantic judgments in the processing of English compound words. We compare static word vectors (GloVe) and contextualized embeddings (BERT) again…
Exponential Family Word Embeddings: An Iterative Approach for Learning Word Vectors
GloVe and Skip-gram word embedding methods learn word vectors by decomposing a denoised matrix of word co-occurrences into a product of low-rank matrices. In this work, we propose an iterative algorithm for computing wor…
Word Embeddings