Building Static Embeddings from Contextual Ones: Is It Useful for Building Distributional Thesauri?
While contextual language models are now dominant in the field of Natural Language Processing, the representations they build at the token level are not always suitable for all uses. In this article, we propose a new method for building word or type-level embeddings from contextual models. This method combines the generalization and the aggregation of token representations. We evaluate it for a large set of English nouns from the perspective of the building of distributional thesauri for extracting semantic similarity relations. Moreover, we analyze the differences between static embeddings and type-level embeddings according to features such as the frequency of words or the type of semantic relations these embeddings account for, showing that the properties of these two types of embeddings can be complementary and exploited for further improving distributional thesauri.
Code (0)
등록된 구현이 없습니다.
Tasks
Semantic SimilaritySemantic Textual SimilarityVocal Bursts Type PredictionSimilar Papers 제목 키워드 기반
Dynamic Contextualized Word Embeddings
Static word embeddings that represent words by a single vector cannot capture the variability of word meaning in different linguistic and extralinguistic contexts. Building on prior work on contextualized and dynamic wor…
Language ModelingLanguage ModellingWord EmbeddingsCombining static and contextualised multilingual embeddings
Static and contextual multilingual embeddings have complementary strengths. Static embeddings, while less expressive than contextual language models, can be more straightforwardly aligned across multiple languages. Conte…
RetrievalXLM-RSimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings
Word alignments are useful for tasks like statistical and neural machine translation (NMT) and cross-lingual annotation projection. Statistical word aligners perform well, as do methods that extract alignments jointly wi…
Machine TranslationMultilingual Word EmbeddingsNMTTranslation+2Caveats of Measuring Semantic Change of Cognates and Borrowings using Multilingual Word Embeddings
Cognates and borrowings carry different aspects of etymological evolution. In this work, we study semantic change of such items using multilingual word embeddings, both static and contextualised. We underline caveats ide…
Multilingual Word EmbeddingsWord EmbeddingsCombining Static and Contextualised Multilingual Embeddings
Static and contextual multilingual embeddings have complementary strengths. Static embeddings, while less expressive than contextual language models, can be more straightforwardly aligned across multiple languages. We co…
RetrievalXLM-R