Locating Language-Specific Information in Contextualized Embeddings
Multilingual pretrained language models (MPLMs) exhibit multilinguality and are well suited for transfer across languages. Most MPLMs are trained in an unsupervised fashion and the relationship between their objective and multilinguality is unclear. More specifically, the question whether MPLM representations are language-agnostic or they simply interleave well with learned task prediction heads arises. In this work, we locate language-specific information in MPLMs and identify its dimensionality and the layers where this information occurs. We show that language-specific information is scattered across many dimensions, which can be projected into a linear subspace. Our study contributes to a better understanding of MPLM representations, going beyond treating them as unanalyzable blobs of information.
Code (1)
Similar Papers 제목 키워드 기반
On the Cross-lingual Transferability of Contextualized Sense Embeddings
In this paper we analyze the extent to which contextualized sense embeddings, i.e., sense embeddings that are computed based on contextualized word embeddings, are transferable across languages.To this end, we compiled a…
Word EmbeddingsWord Sense DisambiguationLeveraging Three Types of Embeddings from Masked Language Models in Idiom Token Classification
Many linguistic expressions have idiomatic and literal interpretations, and the automatic distinction of these two interpretations has been studied for decades. Recent research has shown that contextualized word embeddin…
Classificationtoken-classificationToken ClassificationWord EmbeddingsWhat Does This Word Mean? Explaining Contextualized Embeddings with Natural Language Definition
Contextualized word embeddings have boosted many NLP tasks compared with traditional static word embeddings. However, the word with a specific sense may have different contextualized embeddings due to its various context…
Word EmbeddingsRepresenting Affect Information in Word Embeddings
A growing body of research in natural language processing (NLP) and natural language understanding (NLU) is investigating human-like knowledge learned or encoded in the word embeddings from large language models. This is…
Natural Language UnderstandingWord EmbeddingsClinical Named Entity Recognition using Contextualized Token Representations
The clinical named entity recognition (CNER) task seeks to locate and classify clinical terminologies into predefined categories, such as diagnostic procedure, disease disorder, severity, medication, medication dosage, a…
DiagnosticLanguage Modellingnamed-entity-recognitionNamed Entity Recognition+2