Extrapolating Multilingual Understanding Models as Multilingual Generators
Multilingual understanding models (or encoder-based), pre-trained via masked language modeling, have achieved promising results on many language understanding tasks (e.g., mBERT). However, these non-autoregressive (NAR) models still struggle to generate high-quality texts compared with autoregressive (AR) models. Considering that encoder-based models have the advantage of efficient generation and self-correction abilities, this paper explores methods to empower multilingual understanding models the generation abilities to get a unified model. Specifically, we start from a multilingual encoder (XLM-R) and propose a \textbf{S}emantic-\textbf{G}uided \textbf{A}lignment-then-Denoising (SGA) approach to adapt an encoder to a multilingual generator with a small number of new parameters. Experiments show that the proposed approach is an effective adaption method, outperforming widely-used initialization-based methods with gains of 9.4 BLEU on machine translation, 8.1 Rouge-L on question generation, and 5.5 METEOR on story generation on XLM-R$_{large}$. On the other hand, we observe that XLM-R is still inferior to mBART in supervised settings despite better results on zero-shot settings, indicating that more exploration is required to make understanding models strong generators.
Code (0)
등록된 구현이 없습니다.
Tasks
DenoisingLanguage ModelingLanguage ModellingMachine TranslationMasked Language ModelingQuestion GenerationQuestion-GenerationStory GenerationXLM-RMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Authorship Attribution in Multilingual Machine-Generated Texts
As Large Language Models (LLMs) have reached human-like fluency and coherence, distinguishing machine-generated text (MGT) from human-written content becomes increasingly difficult. While early efforts in MGT detection h…
Binary ClassificationIRCoder: Intermediate Representations Make Language Models Robust Multilingual Code Generators
Code understanding and generation have fast become some of the most popular applications of language models (LMs). Nonetheless, research on multilingual aspects of Code-LMs (i.e., LMs for code generation) such as cross-l…
Code CompletionCode GenerationCross-Lingual TransferData Augmentation+2Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification
Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, making them promising tools in both high- and low-resource languages. One particularly valuable use case is generating synthetic sample…
Synthetic Data GenerationInvestigating Language Preference of Multilingual RAG Systems
Multilingual Retrieval-Augmented Generation (mRAG) systems enhance language models by integrating external multilingual information to produce context-aware responses. However, mRAG systems struggle with retrieving relev…
RAGRetrievalRetrieval-augmented GenerationMULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark
There is a lack of research into capabilities of recent LLMs to generate convincing text in languages other than English and into performance of detectors of machine-generated text in multilingual settings. This is also …
Benchmarkingde-enText Detection