Multilingual BERT Post-Pretraining Alignment

2020-10-23 · NAACL 2021 4 · Lin Pan, Chung-Wei Hang, Haode Qi, Abhishek Shah, Saloni Potdar, Mo Yu

We propose a simple method to align multilingual contextual embeddings as a post-pretraining step for improved zero-shot cross-lingual transferability of the pretrained models. Using parallel data, our method aligns embeddings on the word level through the recently proposed Translation Language Modeling objective as well as on the sentence level via contrastive learning and random input shuffling. We also perform sentence-level code-switching with English when finetuning on downstream tasks. On XNLI, our best model (initialized from mBERT) improves over mBERT by 4.7% in the zero-shot setting and achieves comparable result to XLM for translate-train while using less than 18% of the same parallel data and 31% less model parameters. On MLQA, our model outperforms XLM-R_Base that has 57% more parameters than ours.

📄 PDF Abstract BibTeX arXiv:2010.12547

Code (0)

등록된 구현이 없습니다.

Tasks

Contrastive LearningLanguage ModelingLanguage ModellingSentenceTranslation

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

mBERT mBERT

Contrastive Learning 설명 없음

Attention Dropout Attention Dropout is a type of dropout used in attention-based architectures, where elements are randomly dropped out of the…

Residual Connection 설명 없음

Attention 설명 없음

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Adam 설명 없음

Identifying Necessary Elements for BERT's Multilinguality

2020-05-01 · Philipp Dufter, Hinrich Schütze

It has been shown that multilingual BERT (mBERT) yields high quality multilingual representations and enables effective zero-shot transfer. This is surprising given that mBERT does not use any crosslingual signal during …

Identifying Elements Essential for BERT's Multilinguality

2020-11-01 · EMNLP 2020 11 · Philipp Dufter, Hinrich Sch{\"u}tze

Improved Multilingual Language Model Pretraining for Social Media Text via Translation Pair Prediction

2021-10-20 · WNUT (ACL) 2021 11 · Shubhanshu Mishra, Aria Haghighi

We evaluate a simple approach to improving zero-shot multilingual transfer of mBERT on social media corpus by adding a pretraining task called translation pair prediction (TPP), which predicts whether a pair of cross-lin…

Benchmarking Language Modeling Language Modelling NER+7

A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT

2020-04-29 · EMNLP 2020 11 · Masaaki Nagata, Chousa Katsuki, Masaaki Nishino

We present a novel supervised word alignment method based on cross-language span prediction. We first formalize a word alignment problem as a collection of independent predictions from a token in the source sentence to a…

Question Answering Sentence Word Alignment

RUBERT: A Bilingual Roman Urdu BERT Using Cross Lingual Transfer Learning

2021-02-22 · Usama Khalid, Mirza Omer Beg, Muhammad Umair Arshad

In recent studies, it has been shown that Multilingual language models underperform their monolingual counterparts. It is also a well-known fact that training and maintaining monolingual models for each language is a cos…

Cross-Lingual Transfer Transfer Learning