Combining Static and Contextualised Multilingual Embeddings
Static and contextual multilingual embeddings have complementary strengths. Static embeddings, while less expressive than contextual language models, can be more straightforwardly aligned across multiple languages. We combine the strengths of static and contextual models to improve multilingual representations. We extract static embeddings for 40 languages from XLM-R, validate those embeddings with cross-lingual word retrieval, and then align them using VecMap. This results in high-quality, highly multilingual static embeddings. Then we apply a novel continued pre-training approach to XLM-R, leveraging the high quality alignment of our static embeddings to better align the representation space of XLM-R. We show positive results for multiple complex semantic tasks. We release the static embeddings and the continued pre-training code. Unlike most previous work, our continued pre-training approach does not require parallel text.
Code (1)
Tasks
RetrievalXLM-RMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Combining static and contextualised multilingual embeddings
Static and contextual multilingual embeddings have complementary strengths. Static embeddings, while less expressive than contextual language models, can be more straightforwardly aligned across multiple languages. Conte…
RetrievalXLM-RCaveats of Measuring Semantic Change of Cognates and Borrowings using Multilingual Word Embeddings
Cognates and borrowings carry different aspects of etymological evolution. In this work, we study semantic change of such items using multilingual word embeddings, both static and contextualised. We underline caveats ide…
Multilingual Word EmbeddingsWord EmbeddingsLearning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy
Contextualised word embeddings generated from Neural Language Models (NLMs), such as BERT, represent a word with a vector that considers the semantics of the target word as well its context. On the other hand, static wor…
Word EmbeddingsWord Sense DisambiguationDebiasing Pre-trained Contextualised Embeddings
In comparison to the numerous debiasing methods proposed for the static non-contextualised word embeddings, the discriminative biases in contextualised embeddings have received relatively little attention. We propose a f…
SentenceWord Embeddings