XLM

2000년 도입 · 논문 57편에서 사용

XLM is a Transformer based architecture that is pre-trained using one of three language modelling objectives: 1. Causal Language Modeling - models the probability of a word given the previous words in a sentence. 2. Masked Language Modeling - the masked language modeling objective of BERT. 3. Translation Language Modeling - a (new) translation language modeling objective for improving cross-lingual pre-training. The authors find that both the CLM and MLM approaches provide strong cross-lingual features that can be used for pretraining models.

출처: Cross-lingual Language Model Pretraining

소개 논문: Cross-lingual Language Model Pretraining