Strengthening Target-Language Features: SAE-Based Steering for Multilingual Inference

2026-08-05 · Hongsheng Wang, Philipp Koehn arxiv

Multilingual large language models exhibit substantial performance differences across languages, while existing adaptation methods often require parameter updates and considerable multilingual training data. We propose an inference-time multilingual steering method that uses pretrained sparse autoencoders to identify and strengthen target-language-related features. Using multilingual parallel sentences, we compare SAE activations across languages and select a small number of layer-specific features associated with each target language. These features are decoded into steering signals and injected into the model's hidden states without additional training. Experiments with Gemma-3-12B-it show average accuracy improvements of 10.9 percentage points on XCOPA, 5.3 points on XNLI, and 1.9 points on MGSM.

📄 PDF Abstract BibTeX arXiv:2608.04904

Code (0)

등록된 구현이 없습니다.

Causal Language Control in Multilingual Transformers via Sparse Feature Steering

2025-07-17 · Cheng-Ting Chou, George Liu, Jessica Sun, Cole Blondin 외 arxiv

Deterministically controlling the target generation language of large multilingual language models (LLMs) remains a fundamental challenge, particularly in zero-shot settings where neither explicit language prompts nor fi…

LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering

2026-04-04 · Sing Hieng Wong, Hassan Sajjad, A. B. Siddique arxiv

Large language models (LLMs) show strong multilingual capabilities, yet reliably controlling the language of their outputs remains difficult. Representation-level steering addresses this by adding language-specific vecto…

Target-Language Generation in Multilingual Models: Activation Steering and Optimal Control

2026-09-15 · James A. Michaelov, Carmen Amo Alonso, Tyler A. Chang, Roger P. Levy arxiv

Ensuring that multilingual language models generate coherent text in a specific target language is a major issue in multilingual language modeling. We develop an optimal control method for target-language text generation…

Text Generation

Cross-Lingual Steering for Figurative Language Generation

2026-05-28 · Linfeng Liu, Tiffany Zhan, Louie Hong Yao, Saptarshi Ghosh 외 arxiv

Multilingual large language models can generate figurative language, but whether the internal signals driving this behavior are language-specific or reusable across languages is unclear. Using activation steering as a pr…

Cross-Lingual Activation Steering for Multilingual Language Models

2026-01-23 · Rhitabrat Pokharel, Ameeta Agrawal, Tanay Nagar arxiv

Large language models exhibit strong multilingual capabilities, yet significant performance gaps persist between dominant and non-dominant languages. Prior work attributes this gap to imbalances between shared and langua…