mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinations

2023-05-23 · Jonas Pfeiffer, Francesco Piccinno, Massimo Nicosia, Xinyi Wang, Machel Reid, Sebastian Ruder

Multilingual sequence-to-sequence models perform poorly with increased language coverage and fail to consistently generate text in the correct target language in few-shot settings. To address these challenges, we propose mmT5, a modular multilingual sequence-to-sequence model. mmT5 utilizes language-specific modules during pre-training, which disentangle language-specific information from language-agnostic information. We identify representation drift during fine-tuning as a key limitation of modular generative models and develop strategies that enable effective zero-shot transfer. Our model outperforms mT5 at the same parameter sizes by a large margin on representative natural language understanding and generation tasks in 40+ languages. Compared to mT5, mmT5 raises the rate of generating text in the correct language under zero-shot settings from 7% to 99%, thereby greatly alleviating the source language hallucination problem.

📄 PDF Abstract BibTeX arXiv:2305.14224

Code (0)

등록된 구현이 없습니다.

Tasks

HallucinationNatural Language Understanding

Methods 이 논문이 사용한 방법론

Gated Linear Unit A Gated Linear Unit, or GLU computes: $$ \mathrm{GLU}(a, b) = a \otimes \sigma(b) $$ It is used in natural language processing architectures, for example the Gated CNN,…

Attention 설명 없음

fail 설명 없음

Softmax The Softmax output function transforms a previous layer's output into a vector of probabilities. It is commonly used for multiclass classification. Given an input vector $x$…

Layer Normalization Unlike batch normalization, Layer Normalization directly estimates the normalization statistics from the summed inputs…

Inverse Square Root Schedule Inverse Square Root is a learning rate schedule 1 / $\sqrt{\max\left(n, k\right)}$ where $n$ is the current training iteration and $k$ is the number of warm-up steps. This…

BPE Byte Pair Encoding, or BPE, is a subword segmentation algorithm that encodes rare and unknown words as sequences of subword units. The intuition is that various word…

Dropout Dropout is a regularization technique for neural networks that drops a unit (along with connections) at training time with a specified probability $p$ (a common value is…

Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment

2024-07-20 · Yongxin Huang, Kexin Wang, Goran Glavaš, Iryna Gurevych

Multilingual sentence encoders are commonly obtained by training multilingual language models to map sentences from different languages into a shared semantic space. As such, they are subject to curse of multilinguality,…

Contrastive Learning Multiple-choice Sentence Sentence Embeddings+1

The Obscure Limitation of Modular Multilingual Language Models

2023-11-21 · Muhammad Farid Adilazuarda, Samuel Cahyawijaya, Ayu Purwarianti

We expose the limitation of modular multilingual language models (MLMs) in multilingual inference scenarios with unknown languages. Existing evaluations of modular MLMs exclude the involvement of language identification …

Language Identification

Master-ASR: Achieving Multilingual Scalability and Low-Resource Adaptation in ASR with Modular Learning

2023-06-23 · Zhongzhi Yu, Yang Zhang, Kaizhi Qian, Yonggan Fu 외

Despite the impressive performance recently achieved by automatic speech recognition (ASR), we observe two primary challenges that hinder its broader applications: (1) The difficulty of introducing scalability into the m…

Automatic Speech Recognition Automatic Speech Recognition (ASR)speech-recognition Speech Recognition+1

ColBERT-XM: A Modular Multi-Vector Representation Model for Zero-Shot Multilingual Information Retrieval

2024-02-23 · Antoine Louis, Vageesh Saxena, Gijs Van Dijck, Gerasimos Spanakis

State-of-the-art neural retrievers predominantly focus on high-resource languages like English, which impedes their adoption in retrieval scenarios involving other languages. Current approaches circumvent the lack of hig…

Cross-Lingual Transfer Information Retrieval Retrieval

Mixture of Modular Experts: Distilling Knowledge from a Multilingual Teacher into Specialized Modular Language Models

2024-07-28 · Mohammed Al-Maamari, Mehdi Ben Amor, Michael Granitzer

This research combines Knowledge Distillation (KD) and Mixture of Experts (MoE) to develop modular, efficient multilingual language models. Key objectives include evaluating adaptive versus fixed alpha methods in KD and …

Knowledge Distillation Mixture-of-Experts