Papers XLM-R
“XLM-R” 태그가 달린 논문 221편 · 필터 해제
QASiNa: Religious Domain Question Answering using Sirah Nabawiyah
Nowadays, Question Answering (QA) tasks receive significant research focus, particularly with the development of Large Language Model (LLM) such as Chat GPT [1]. LLM can be applied to various domains, but it contradicts …
Language ModellingLarge Language ModelQuestion AnsweringXLM-RExploring the Maze of Multilingual Modeling
Multilingual language models have gained significant attention in recent years, enabling the development of applications that meet diverse linguistic contexts. In this paper, we present a comprehensive evaluation of thre…
Language ModellingModel SelectionPretrained Multilingual Language Modelstext-classification+3Mixed-Distil-BERT: Code-mixed Language Modeling for Bangla, English, and Hindi
One of the most popular downstream tasks in the field of Natural Language Processing is text classification. Text classification tasks have become more daunting when the texts are code-mixed. Though they are not exposed …
Language ModelingLanguage Modellingtext-classificationText Classification+1Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models
Language models such as mBERT, XLM-R, and BLOOM aim to achieve multilingual generalization or compression to facilitate transfer to a large number of (potentially unseen) languages. However, these models should ideally a…
FairnessXLM-RSelf-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models
We investigate the effects of post-training quantization and quantization-aware training on the generalization of Transformer language models. We present a new method called self-distilled quantization (SDQ) that minimiz…
QuantizationXLM-RDoes mBERT understand Romansh? Evaluating word embeddings using word alignment
We test similarity-based word alignment models (SimAlign and awesome-align) in combination with word embeddings from mBERT and XLM-R on parallel sentences in German and Romansh. Since Romansh is an unseen language, we ar…
SentenceWord AlignmentWord EmbeddingsXLM-RXSemPLR: Cross-Lingual Semantic Parsing in Multiple Natural Languages and Meaning Representations
Cross-Lingual Semantic Parsing (CLSP) aims to translate queries in multiple natural languages (NLs) into meaning representations (MRs) such as SQL, lambda calculus, and logic forms. However, existing CLSP models are sepa…
Cross-Lingual TransferDecoderSemantic ParsingTransfer Learning+1Exploring the Relationship between Alignment and Cross-lingual Transfer in Multilingual Transformers
Without any explicit cross-lingual training data, multilingual language models can achieve cross-lingual transfer. One common way to improve this transfer is to perform realignment steps before fine-tuning, i.e., to trai…
Cross-Lingual TransferPOSPOS TaggingXLM-RDistilling Efficient Language-Specific Models for Cross-Lingual Transfer
Massively multilingual Transformers (MMTs), such as mBERT and XLM-R, are widely used for cross-lingual transfer learning. While these are pretrained to represent hundreds of languages, end users of NLP systems are often …
Cross-Lingual TransferTransfer LearningXLM-RZero-Shot Cross-Lingual TransferReadMe++: Benchmarking Multilingual Language Models for Multi-Domain Readability Assessment
We present a comprehensive evaluation of large language models for multilingual readability assessment. Existing evaluation resources lack domain and language diversity, limiting the ability for cross-domain and cross-li…
BenchmarkingCross-Lingual TransferDiversityDomain Generalization+2FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models
Using model weights pretrained on a high-resource language as a warm start can reduce the need for data and compute to obtain high-quality language models for other, especially low-resource, languages. However, if we wan…
Language ModelingLanguage ModellingNERSemantic Similarity+2Extrapolating Multilingual Understanding Models as Multilingual Generators
Multilingual understanding models (or encoder-based), pre-trained via masked language modeling, have achieved promising results on many language understanding tasks (e.g., mBERT). However, these non-autoregressive (NAR) …
DenoisingLanguage ModelingLanguage ModellingMachine Translation+5DUMB: A Benchmark for Smart Evaluation of Dutch Models
We introduce the Dutch Model Benchmark: DUMB. The benchmark includes a diverse set of datasets for low-, medium- and high-resource tasks. The total set of nine tasks includes four tasks that were previously not available…
XLM-RGlot500: Scaling Multilingual Corpora and Language Models to 500 Languages
The NLP community has mainly focused on scaling Large Language Models (LLMs) vertically, i.e., making them better for about 100 languages. We instead scale LLMs horizontally: we create, through continued pretraining, Glo…
Language ModellingXLM-RESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain
The increasing number of benchmarks for Natural Language Processing (NLP) tasks in the computational job market domain highlights the demand for methods that can handle job-related tasks such as skill extraction, skill c…
De-identificationLanguage ModelingLanguage ModellingMasked Language Modeling+1DN at SemEval-2023 Task 12: Low-Resource Language Text Classification via Multilingual Pretrained Language Model Fine-tuning
In recent years, sentiment analysis has gained significant importance in natural language processing. However, most existing models and datasets for sentiment analysis are developed for high-resource languages, such as E…
Language ModelingLanguage ModellingSentiment Analysistext-classification+2USTC-NELSLIP at SemEval-2023 Task 2: Statistical Construction and Dual Adaptation of Gazetteer for Multilingual Complex NER
This paper describes the system developed by the USTC-NELSLIP team for SemEval-2023 Task 2 Multilingual Complex Named Entity Recognition (MultiCoNER II). A method named Statistical Construction and Dual Adaptation of Gaz…
named-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)NER+3Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroese
Multilingual language models have pushed state-of-the-art in cross-lingual NLP transfer. The majority of zero-shot cross-lingual transfer, however, use one and the same massively multilingual transformer (e.g., mBERT or …
Cross-Lingual Transfernamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+5GreekBART: The First Pretrained Greek Sequence-to-Sequence Model
The era of transfer learning has revolutionized the fields of Computer Vision and Natural Language Processing, bringing powerful pretrained models with exceptional performance across a variety of tasks. Specifically, Nat…
Natural Language InferenceText GenerationTransfer LearningXLM-RTollywood Emotions: Annotation of Valence-Arousal in Telugu Song Lyrics
Emotion recognition from a given music track has heavily relied on acoustic features, social tags, and metadata but is seldom focused on lyrics. There are no datasets of Indian language songs that contain both valence an…
Emotion RecognitionMusic Emotion RecognitionXLM-R