Specializing Multilingual Language Models: An Empirical Study
Pretrained multilingual language models have become a common tool in transferring NLP capabilities to low-resource languages, often with adaptations. In this work, we study the performance, extensibility, and interaction of two such adaptations: vocabulary augmentation and script transliteration. Our evaluations on part-of-speech tagging, universal dependency parsing, and named entity recognition in nine diverse low-resource languages uphold the viability of these approaches while raising new questions around how to optimally adapt multilingual models to low-resource settings.
Code (1)
Tasks
Dependency Parsingnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)Part-Of-Speech TaggingPretrained Multilingual Language ModelsTransliterationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A dual task learning approach to fine-tune a multilingual semantic speech encoder for Spoken Language Understanding
Self-Supervised Learning is vastly used to efficiently represent speech for Spoken Language Understanding, gradually replacing conventional approaches. Meanwhile, textual SSL models are proposed to encode language-agnost…
Self-Supervised LearningSpoken Language UnderstandingSemantic enrichment towards efficient speech representations
Over the past few years, self-supervised learned speech representations have emerged as fruitful replacements for conventional surface representations when solving Spoken Language Understanding (SLU) tasks. Simultaneousl…
Spoken Language UnderstandingEmbedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages
Pre-trained multilingual language models underpin a large portion of modern NLP tools outside of English. A strong baseline for specializing these models for specific languages is Language-Adaptive Pre-Training (LAPT). H…
Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
While general-purpose large language models (LLMs) demonstrate proficiency on multiple tasks within the domain of translation, approaches based on open LLMs are competitive only when specializing on a single task. In thi…
Language ModelingLanguage ModellingLarge Language ModelTranslationSpecializing Pre-trained Language Models for Better Relational Reasoning via Network Pruning
Pretrained masked language models (PLMs) were shown to be inheriting a considerable amount of relational knowledge from the source corpora. In this paper, we present an in-depth and comprehensive study concerning special…
Network PruningRelational Reasoning