Papers XLM-R
“XLM-R” 태그가 달린 논문 221편 · 필터 해제
Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure
Multilingual pre-trained language models, such as mBERT and XLM-R, have shown impressive cross-lingual ability. Surprisingly, both of them use multilingual masked language model (MLM) without any cross-lingual supervisio…
Language ModelingLanguage ModellingNatural Language InferenceRetrieval+3Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages
Pre-trained multilingual language models such as mBERT and XLM-R have demonstrated great potential for zero-shot cross-lingual transfer to low web-resource languages (LRL). However, due to limited model capacity, the lar…
Cross-Lingual TransferXLM-RZero-Shot Cross-Lingual TransferTransfer Learning Approaches for Building Cross-Language Dense Retrieval Models
The advent of transformer-based models such as BERT has led to the rise of neural ranking models. These models have improved the effectiveness of retrieval systems well beyond that of lexical term matching models such as…
Document RankingInformation RetrievalRetrievalTransfer Learning+1Are Pretrained Multilingual Models Equally Fair Across Languages?
Pretrained multilingual language models can help bridge the digital language divide, enabling high-quality NLP models for lower-resourced languages. Studies of multilingual models have so far focused on performance, cons…
Cloze TestFairnessPretrained Multilingual Language ModelsXLM-RLexicon based Fine-tuning of Multilingual Language Models for Sentiment Analysis of Low-resource Languages
Massively multilingual language models (MMLM) such as mBERT and XLM-R have shown good cross-lingual transferability. However, they are not specifically trained to capture cross-lingual signals with respect to sentiment w…
ClassificationSentiment AnalysisSentiment ClassificationXLM-RFrustratingly Simple Regularization to Improve Zero-shot Cross-lingual Robustness
Large-scale multilingual pretrained encoders, such as mBERT and XLM-R, have demonstrated impressive zero-shot cross-lingual transfer capability across multiple NLP tasks. However, as we show in this paper, these models s…
Cross-Lingual TransferXLM-RZero-Shot Cross-Lingual TransferThe Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer
Large pre-trained multilingual models such as mBERT and XLM-R enabled effective cross-lingual zero-shot transfer in many NLP tasks. A cross-lingual adjustment of these models using a small parallel corpus can further imp…
Machine TranslationNERXLM-RZeroBERTo: Leveraging Zero-Shot Text Classification by Topic Modeling
Traditional text classification approaches often require a good amount of labeled data, which is difficult to obtain, especially in restricted domains or less widespread languages. This lack of labeled data has led to th…
Classificationtext-classificationText ClassificationXLM-R+2CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domain
The field of natural language processing (NLP) has recently seen a large change towards using pre-trained language models for solving almost any task. Despite showing great improvements in benchmark datasets for various …
Clinical Concept ExtractionLanguage ModellingSentenceXLM-RMultilingual Pre-training with Universal Dependency Learning
The pre-trained language model (PrLM) demonstrates domination in downstream natural language processing tasks, in which multilingual PrLM takes advantage of language universality to alleviate the issue of limited resourc…
Dependency ParsingLanguage ModelingLanguage ModellingNatural Language Understanding+1DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-traini…
Language ModelingLanguage ModellingNatural Language InferenceNatural Language Understanding+2Gradient Sparsification For \emph{Masked Fine-Tuning} of Transformers
Fine-tuning masked language models is widely adopted for transfer learning to downstream tasks and can be achieved by (1) freezing gradients of the pretrained network or only updating gradients of a newly added classific…
Transfer LearningXLM-RCombining static and contextualised multilingual embeddings
Static and contextual multilingual embeddings have complementary strengths. Static embeddings, while less expressive than contextual language models, can be more straightforwardly aligned across multiple languages. Conte…
RetrievalXLM-RMulti-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching
Previous studies have proved that cross-lingual knowledge distillation can significantly improve the performance of pre-trained models for cross-lingual similarity matching tasks. However, the student model needs to be l…
Contrastive LearningKnowledge DistillationSemantic SimilaritySemantic Textual Similarity+1Prix-LM: Pretraining for Multilingual Knowledge Base Construction
Knowledge bases (KBs) contain plenty of structured world and commonsense knowledge. As such, they often complement distributional text-based information and facilitate various downstream tasks. Since their manual constru…
Bilingual Lexicon InductionCausal Language ModelingCross-Lingual Entity LinkingEntity Linking+5Saliency-based Multi-View Mixed Language Training for Zero-shot Cross-lingual Classification
Recent multilingual pre-trained models, like XLM-RoBERTa (XLM-R), have been demonstrated effective in many cross-lingual tasks. However, there are still gaps between the contextualized representations of similar words in…
Cross-Lingual Sentiment ClassificationDialogue State TrackingMULTI-VIEW LEARNINGSentiment Analysis+2NICT Kyoto Submission for the WMT’21 Quality Estimation Task: Multimetric Multilingual Pretraining for Critical Error Detection
This paper presents the NICT Kyoto submission for the WMT’21 Quality Estimation (QE) Critical Error Detection shared task (Task 3). Our approach relies mainly on QE model pretraining for which we used 11 language pairs, …
Language ModelingLanguage ModellingMasked Language ModelingSentence+2IndoNLI: A Natural Language Inference Dataset for Indonesian
We present IndoNLI, the first human-elicited NLI dataset for Indonesian. We adapt the data collection protocol for MNLI and collect nearly 18K sentence pairs annotated by crowd workers and experts. The expert-annotated d…
Natural Language InferenceSentenceSpatial ReasoningXLM-RTowards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
This paper demonstrates that multilingual pretraining and multilingual fine-tuning are both critical for facilitating cross-lingual transfer in zero-shot translation, where the neural machine translation (NMT) model is t…
Abstractive Text SummarizationCross-Lingual Abstractive SummarizationCross-Lingual TransferDecoder+4Prix-LM: Pretraining for Multilingual Knowledge Base Construction
Knowledge bases (KBs) contain plenty of structured world and commonsense knowledge. As such, they often complement distributional text-based information and facilitate various downstream tasks. Since their manual constru…
Bilingual Lexicon InductionCausal Language ModelingCross-Lingual Entity LinkingEntity Linking+5