On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons

2024-04-03 · Takeshi Kojima, Itsuki Okimura, Yusuke Iwasawa, Hitomi Yanaka, Yutaka Matsuo

Current decoder-based pre-trained language models (PLMs) successfully demonstrate multilingual capabilities. However, it is unclear how these models handle multilingualism. We analyze the neuron-level internal behavior of multilingual decoder-based PLMs, Specifically examining the existence of neurons that fire ``uniquely for each language'' within decoder-only multilingual PLMs. We analyze six languages: English, German, French, Spanish, Chinese, and Japanese, and show that language-specific neurons are unique, with a slight overlap (< 5%) between languages. These neurons are mainly distributed in the models' first and last few layers. This trend remains consistent across languages and models. Additionally, we tamper with less than 1% of the total neurons in each model during inference and demonstrate that tampering with a few language-specific neurons drastically changes the probability of target language occurrence in text generation.

📄 PDF Abstract BibTeX arXiv:2404.02431

Code (1)

kojima-takeshi188/lang_neuron 공식 구현 pytorch

Tasks

Decoder Text Generation

What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representations

2025-07-27 · Katharina Trinley, Toshiki Nakai, Tatiana Anikina, Tanja Baeumel arxiv

Large language models (LLMs) excel at multilingual tasks, yet their internal language processing remains poorly understood. We analyze how Aya-23-8B, a decoder-only LLM trained on balanced multilingual data, handles code…

Cross-Lingual Transfer

Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality

2026-03-18 · Mengyu Bu, Yang Feng arxiv

Large language models (LLMs) exhibit strong general intelligence, yet their multilingual performance remains highly imbalanced. Although LLMs encode substantial cross-lingual knowledge in a unified semantic space, they o…

General Knowledge

Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks

2024-06-19 · Dan Saattrup Nielsen, Kenneth Enevoldsen, Peter Schneider-Kamp

This paper explores the performance of encoder and decoder language models on multilingual Natural Language Understanding (NLU) tasks, with a broad focus on Germanic languages. Building upon the ScandEval benchmark, init…

Decoder Language Modeling Language Modelling Model Selection+1

DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders

2021-06-25 · Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang 외

While pretrained encoders have achieved success in various natural language understanding (NLU) tasks, there is a gap between these pretrained encoders and natural language generation (NLG). NLG tasks are often based on …

Abstractive Text Summarization Decoder Machine Translation Natural Language Understanding+5

Multilingual Unsupervised NMT using Shared Encoder and Language-Specific Decoders

2019-07-01 · ACL 2019 7 · Sukanta Sen, Kamal Kumar Gupta, Asif Ekbal, Pushpak Bhattacharyya

In this paper, we propose a multilingual unsupervised NMT scheme which jointly trains multiple languages with a shared encoder and multiple decoders. Our approach is based on denoising autoencoding of each language and b…

Denoising NMT Translation