Estimating word co-occurrence probabilities from pretrained static embeddings using a log-bilinear model
We investigate how to use pretrained static word embeddings to deliver improved estimates of bilexical co-occurrence probabilities: conditional probabilities of one word given a single other word in a specific relationship. Such probabilities play important roles in psycholinguistics, corpus linguistics, and usage-based cognitive modeling of language more generally. We propose a log-bilinear model taking pretrained vector representations of the two words as input, enabling generalization based on the distributional information contained in both vectors. We show that this model outperforms baselines in estimating probabilities of adjectives given nouns that they attributively modify, and probabilities of nominal direct objects given their head verbs, given limited training data in Arabic, English, Korean, and Spanish.
Code (0)
등록된 구현이 없습니다.
Tasks
Word EmbeddingsSimilar Papers 제목 키워드 기반
On the Interpretability and Significance of Bias Metrics in Texts: a PMI-based Approach
In recent years, word embeddings have been widely used to measure biases in texts. Even if they have proven to be effective in detecting a wide variety of biases, metrics based on word embeddings lack transparency and in…
Word EmbeddingsOn the interpretability and significance of bias metrics in texts: a PMI-based approach
In recent years, the use of word embeddings has become popular to measure the presence of biases in texts. Despite the fact that these measures have been proven to be effective in detecting a wide variety of biases, metr…
Word EmbeddingsStreaming word similarity mining on the cheap
Accurately and efficiently estimating word similarities from text is fundamental in natural language processing. In this paper, we propose a fast and lightweight method for estimating similarities from streams by explici…
Document ClassificationWord AlignmentWord EmbeddingsWord SimilaritySemGloVe: Semantic Co-occurrences for GloVe from BERT
GloVe learns word embeddings by leveraging statistical information from word co-occurrence matrices. However, word pairs in the matrices are extracted from a predefined local context window, which might lead to limited w…
Language ModelingLanguage ModellingWord EmbeddingsWord SimilarityRecognize Foreign Low-Frequency Words with Similar Pairs
Low-frequency words place a major challenge for automatic speech recognition (ASR). The probabilities of these words, which are often important name entities, are generally under-estimated by the language model (LM) due …
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Language ModelingLanguage Modelling+2