Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategy
This paper addresses the challenges in developing language models for less-represented languages, with a focus on Luxembourgish. Despite its active development, Luxembourgish faces a digital data scarcity, exacerbated by Luxembourg's multilingual context. We propose a novel text generation model based on the T5 architecture, combining limited Luxembourgish data with equal amounts, in terms of size and type, of German and French data. We hypothesise that a model trained on Luxembourgish, German, and French will improve the model's cross-lingual transfer learning capabilities and outperform monolingual and large multilingual models. To verify this, the study at hand explores whether multilingual or monolingual training is more beneficial for Luxembourgish language generation. For the evaluation, we introduce LuxGen, a text generation benchmark that is the first of its kind for Luxembourgish.
Code (0)
등록된 구현이 없습니다.
Tasks
Cross-Lingual TransferText GenerationTransfer LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
LuxIT: A Luxembourgish Instruction Tuning Dataset from Monolingual Seed Data
The effectiveness of instruction-tuned Large Language Models (LLMs) is often limited in low-resource linguistic settings due to a lack of high-quality training data. We introduce LuxIT, a novel, monolingual instruction t…
LuxBank: The First Universal Dependency Treebank for Luxembourgish
The Universal Dependencies (UD) project has significantly expanded linguistic coverage across 161 languages, yet Luxembourgish, a West Germanic language spoken by approximately 400,000 people, has remained absent until n…
Adapting Multilingual Embedding Models to Historical Luxembourgish
The growing volume of digitized historical texts requires effective semantic search using text embeddings. However, pre-trained multilingual models, typically evaluated on contemporary texts, face challenges with histori…
ArticlesOptical Character Recognition (OCR)Neural Text Normalization for Luxembourgish using Real-Life Variation Data
Orthographic variation is very common in Luxembourgish texts due to the absence of a fully-fledged standard variety. Additionally, developing NLP tools for Luxembourgish is a difficult task given the lack of annotated an…
Text NormalizationLuxemBERT: Simple and Practical Data Augmentation in Language Model Pre-Training for Luxembourgish
Pre-trained Language Models such as BERT have become ubiquitous in NLP where they have achieved state-of-the-art performance in most NLP tasks. While these models are readily available for English and other widely spoken…
Data AugmentationLanguage ModelingLanguage Modelling