What makes multilingual BERT multilingual?

2020-10-20 · Chi-Liang Liu, Tsung-Yuan Hsu, Yung-Sung Chuang, Hung-Yi Lee

Recently, multilingual BERT works remarkably well on cross-lingual transfer tasks, superior to static non-contextualized word embeddings. In this work, we provide an in-depth experimental study to supplement the existing literature of cross-lingual ability. We compare the cross-lingual ability of non-contextualized and contextualized representation model with the same data. We found that datasize and context window size are crucial factors to the transferability.

📄 PDF Abstract BibTeX arXiv:2010.10938

Code (0)

등록된 구현이 없습니다.

Tasks

Cross-Lingual TransferWord Embeddings

Methods 이 논문이 사용한 방법론

Linear Layer A Linear Layer is a projection $\mathbf{XW + b}$.

Adam 설명 없음

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Dense Connections Dense Connections, or Fully Connected Connections, are a type of layer in a deep neural network that use a linear operation where every input is connected to every output…

Multi-Head Attention 설명 없음

Refunds@Expedia|||How do I get a full refund from Expedia? “How do I get a full refund from Expedia? How do I get a full refund from Expedia? – Call ☎️ +1-(888) 829 (0881) or +1-805-330-4056 or +1-805-330-4056 for Quick Help &…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Linear Warmup With Linear Decay Linear Warmup With Linear Decay is a learning rate schedule in which we increase the learning rate linearly for $n$ updates and then linearly decay afterwards.

To What Degree Can Language Borders Be Blurred In BERT-based Multilingual Spoken Language Understanding?

2020-11-10 · COLING 2020 8 · Quynh Do, Judith Gaspers, Tobias Roding, Melanie Bradford

This paper addresses the question as to what degree a BERT-based multilingual Spoken Language Understanding (SLU) model can transfer knowledge across languages. Through experiments we will show that, although it works su…

Spoken Language Understanding

Identifying Necessary Elements for BERT's Multilinguality

2020-05-01 · Philipp Dufter, Hinrich Schütze

It has been shown that multilingual BERT (mBERT) yields high quality multilingual representations and enables effective zero-shot transfer. This is surprising given that mBERT does not use any crosslingual signal during …

Identifying Elements Essential for BERT's Multilinguality

2020-11-01 · EMNLP 2020 11 · Philipp Dufter, Hinrich Sch{\"u}tze

An Isotropy Analysis in the Multilingual BERT Embedding Space

2021-10-09 · Findings (ACL) 2022 5 · Sara Rajaee, Mohammad Taher Pilehvar

Several studies have explored various advantages of multilingual pre-trained models (such as multilingual BERT) in capturing shared linguistic knowledge. However, less attention has been paid to their limitations. In thi…

Load What You Need: Smaller Versions of Multilingual BERT

2020-10-12 · Amine Abdaoui, Camille Pradel, Grégoire Sigel

Pre-trained Transformer-based models are achieving state-of-the-art results on a variety of Natural Language Processing data sets. However, the size of these models is often a drawback for their deployment in real produc…