How Language-Neutral is Multilingual BERT?
Multilingual BERT (mBERT) provides sentence representations for 104 languages, which are useful for many multi-lingual tasks. Previous work probed the cross-linguality of mBERT using zero-shot transfer learning on morphological and syntactic tasks. We instead focus on the semantic properties of mBERT. We show that mBERT representations can be split into a language-specific component and a language-neutral component, and that the language-neutral component is sufficiently general in terms of modeling semantics to allow high-accuracy word-alignment and sentence retrieval but is not yet good enough for the more difficult task of MT quality estimation. Our work presents interesting challenges which must be solved to build better language-neutral representations, particularly for tasks requiring linguistic transfer of semantics.
Code (1)
Tasks
RetrievalSentenceSentence RetrievalTransfer LearningWord AlignmentMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
On the Language Neutrality of Pre-trained Multilingual Representations
Multilingual contextual embeddings, such as multilingual BERT and XLM-RoBERTa, have proved useful for many multi-lingual tasks. Previous work probed the cross-linguality of the representations indirectly using zero-shot …
Language IdentificationTransfer LearningWord AlignmentVideo Games as a Corpus: Sentiment Analysis using Fallout New Vegas Dialog
We present a method for extracting a multilingual sentiment annotated dialog data set from Fallout New Vegas. The game developers have preannotated every line of dialog in the game in one of the 8 different sentiments: \…
Sentiment AnalysisOn the Language-specificity of Multilingual BERT and the Impact of Fine-tuning
Recent work has shown evidence that the knowledge acquired by multilingual BERT (mBERT) has two components: a language-specific and a language-neutral one. This paper analyses the relationship between them, in the contex…
Language IdentificationNatural Language InferencePOSPOS Tagging+1Cross-neutralising: Probing for joint encoding of linguistic information in multilingual models
Multilingual sentence encoders are widely used to transfer NLP models across languages. The success of this transfer is, however, dependent on the model's ability to encode the patterns of cross-lingual similarity and va…
SentenceXLM-RXED: A Multilingual Dataset for Sentiment Analysis and Emotion Detection
We introduce XED, a multilingual fine-grained emotion dataset. The dataset consists of human-annotated Finnish (25k) and English sentences (30k), as well as projected annotations for 30 additional languages, providing ne…
Sentiment Analysis