Transferring BERT Capabilities from High-Resource to Low-Resource Languages Using Vocabulary Matching
Pre-trained language models have revolutionized the natural language understanding landscape, most notably BERT (Bidirectional Encoder Representations from Transformers). However, a significant challenge remains for low-resource languages, where limited data hinders the effective training of such models. This work presents a novel approach to bridge this gap by transferring BERT capabilities from high-resource to low-resource languages using vocabulary matching. We conduct experiments on the Silesian and Kashubian languages and demonstrate the effectiveness of our approach to improve the performance of BERT models even when the target language has minimal training data. Our results highlight the potential of the proposed technique to effectively train BERT models for low-resource languages, thus democratizing access to advanced language understanding models.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language UnderstandingMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
PCBERT: Parent and Child BERT for Chinese Few-shot NER
Achieving good performance on few-shot or zero-shot datasets has been a long-term challenge for NER. The conventional semantic transfer approaches on NER will decrease model performance when the semantic distribution is …
Few-Shot Learningfew-shot-nerFew-shot NERNERFrom English To Foreign Languages: Transferring Pre-trained Language Models
Pre-trained models have demonstrated their effectiveness in many downstream natural language processing (NLP) tasks. The availability of multilingual pre-trained models enables zero-shot transfer of NLP tasks from high r…
Dependency ParsingGPUNatural Language InferenceCross-lingual transfer of multilingual models on low resource African Languages
Large multilingual models have significantly advanced natural language processing (NLP) research. However, their high resource demands and potential biases from diverse data sources have raised concerns about their effec…
Cross-Lingual TransferCross-Lingual Text Classification with Minimal Resources by Transferring a Sparse Teacher
Cross-lingual text classification alleviates the need for manually labeled documents in a target language by leveraging labeled documents from other languages. Existing approaches for transferring supervision across lang…
General ClassificationRepresentation Learningtext-classificationText ClassificationTransferring Monolingual Model to Low-Resource Language: The Case of Tigrinya
In recent years, transformer models have achieved great success in natural language processing (NLP) tasks. Most of the current state-of-the-art NLP results are achieved by using monolingual transformer models, where the…
Language ModelingLanguage ModellingSentiment AnalysisTransfer Learning