mALBERT: Is a Compact Multilingual BERT Model Still Worth It?
Within the current trend of Pretained Language Models (PLM), emerge more and more criticisms about the ethical andecological impact of such models. In this article, considering these critical remarks, we propose to focus on smallermodels, such as compact models like ALBERT, which are more ecologically virtuous than these PLM. However,PLMs enable huge breakthroughs in Natural Language Processing tasks, such as Spoken and Natural LanguageUnderstanding, classification, Question--Answering tasks. PLMs also have the advantage of being multilingual, and,as far as we know, a multilingual version of compact ALBERT models does not exist. Considering these facts, wepropose the free release of the first version of a multilingual compact ALBERT model, pre-trained using Wikipediadata, which complies with the ethical aspect of such a language model. We also evaluate the model against classicalmultilingual PLMs in classical NLP tasks. Finally, this paper proposes a rare study on the subword tokenizationimpact on language performances.
Code (0)
등록된 구현이 없습니다.
Tasks
Language ModelingLanguage ModellingQuestion AnsweringMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Learning Compact Metrics for MT
Recent developments in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations fr…
Cross-Lingual TransferLanguage ModelingLanguage ModellingMachine Translation+3mHuBERT-147: A Compact Multilingual HuBERT Model
We present mHuBERT-147, the first general-purpose massively multilingual HuBERT speech representation model trained on 90K hours of clean, open-license data. To scale up the multi-iteration HuBERT approach, we use faiss-…
Automatic Speech Recognition (ASR)DiversitymodelSpeech Language Identification+1Measuring Shifts in Attitudes Towards COVID-19 Measures in Belgium Using Multilingual BERT
We classify seven months' worth of Belgian COVID-related Tweets using multilingual BERT and relate them to their governments' COVID measures. We classify Tweets by their stated opinion on Belgian government curfew measur…
EstBERT: A Pretrained Language-Specific BERT for Estonian
This paper presents EstBERT, a large pretrained transformer-based language-specific BERT model for Estonian. Recent work has evaluated multilingual BERT models on Estonian tasks and found them to outperform the baselines…
Morphological Taggingnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+3A Data Bootstrapping Recipe for Low-Resource Multilingual Relation Classification
Relation classification (sometimes called ‘extraction’) requires trustworthy datasets for fine-tuning large language models, as well as for evaluation. Data collection is challenging for Indian languages, because they ar…
ClassificationRelationRelation Classification