The Geometry of Low-Resource Language Representations

2026-08-24 · Francois Meyer, Jan Buys arxiv

The performance gap between low- and high-resource languages in LLMs is widely known, but it remains unclear which internal model factors drive these disparities. In this paper, we characterise this gap through the lens of representational geometry. Comparing the geometric properties of hidden representations across 30 languages reveals that LLM geometry is systematically related to language data availability. The most consistent effect is in final layers, where low-resource languages exhibit representational degeneration. To counter this, we investigate the effectiveness of regularisation terms to penalise degeneration during continued pretraining (CPT). Experiments monolingually adapting 9 base LLMs to 10 African languages show that geometric regularisation successfully reduces representational degeneration during CPT. For larger models, cosine similarity-based regularisation marginally improves performance over vanilla CPT, with more consistent gains on the most challenging tasks. We establish that the representational geometry of low- and high-resource languages in LLMs is measurably distinct, and that targeted geometric intervention is a viable strategy for improving CPT for low-resource languages.

📄 PDF Abstract BibTeX arXiv:2608.23358

Code (0)

등록된 구현이 없습니다.

The Geometry of Multilingual Language Models: An Equality Lens

2023-05-13 · Cheril Shah, Yashashree Chandak, Manan Suri

Understanding the representations of different languages in multilingual language models is essential for comprehending their cross-lingual properties, predicting their performance on downstream tasks, and identifying an…

LAWDR: Language-Agnostic Weighted Document Representations from Pre-trained Models

2021-06-07 · Hongyu Gong, Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán

Cross-lingual document representations enable language understanding in multilingual contexts and allow transfer learning from high-resource to low-resource languages at the document level. Recently large pre-trained lan…

Representation Learning Sentence Sentence Embeddings Transfer Learning

Contrastive Learning for Low Resource Machine Translation

2021-11-16 · ACL ARR November 2021 11 · Anonymous

Representation learning plays a vital role in natural language processing tasks. More recent works study the geometry of the representation space for each layer of pre-trained language models. They find that the context…

Contrastive Learning Data Augmentation Language Modeling Language Modelling+3

LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety

2026-04-13 · Junxiao Yang, Haoran Liu, Jinzhe Tu, Jiale Cheng 외 arxiv

Large language models (LLMs) often demonstrate strong safety performance in high-resource languages, yet exhibit severe vulnerabilities when queried in low-resource languages. We attribute this gap to a mismatch between …

Geometry-Aware Metric Learning for Cross-Lingual Few-Shot Sign Language Recognition on Static Hand Keypoints

2026-03-10 · Chayanin Chamachot, Kanokphan Lertniponphan arxiv

Sign language recognition (SLR) systems typically require large labeled corpora for each language, yet the majority of the world's 300+ sign languages lack sufficient annotated data. Cross-lingual few-shot transfer, pret…

Sign Language Recognition Cross-Lingual Transfer Metric Learning