LightMBERT: A Simple Yet Effective Method for Multilingual BERT Distillation
The multilingual pre-trained language models (e.g, mBERT, XLM and XLM-R) have shown impressive performance on cross-lingual natural language understanding tasks. However, these models are computationally intensive and difficult to be deployed on resource-restricted devices. In this paper, we propose a simple yet effective distillation method (LightMBERT) for transferring the cross-lingual generalization ability of the multilingual BERT to a small student model. The experiment results empirically demonstrate the efficiency and effectiveness of LightMBERT, which is significantly better than the baselines and performs comparable to the teacher mBERT.
Code (0)
등록된 구현이 없습니다.
Tasks
Natural Language UnderstandingXLM-RMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
CAMeMBERT: Cascading Assistant-Mediated Multilingual BERT
Large language models having hundreds of millions, and even billions, of parameters have performed extremely well on a variety of natural language processing (NLP) tasks. Their widespread use and adoption, however, is hi…
Knowledge DistillationLimitations of Knowledge Distillation for Zero-shot Transfer Learning
Pretrained transformer-based encoders such as BERT have been demonstrated to achieve state-of-the-art performance on numerous NLP tasks. Despite their success, BERT style encoders are large in size and have high latency …
CPUCross-Lingual TransferKnowledge DistillationTransfer Learning+1Distilling BERT into Simple Neural Networks with Unlabeled Transfer Data
Recent advances in pre-training huge models on large amounts of text through self supervision have obtained state-of-the-art results in various natural language processing tasks. However, these huge and expensive models …
Knowledge DistillationNERDomain-specific knowledge distillation yields smaller and better models for conversational commerce
We demonstrate that knowledge distillation can be used not only to reduce model size, but to simultaneously adapt a contextual language model to a specific domain. We use Multilingual BERT (mBERT; Devlin et al., 2019) as…
Knowledge DistillationLanguage ModelingLanguage ModellingL3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERT
The multilingual Sentence-BERT (SBERT) models map different languages to common representation space and are useful for cross-language similarity and mining tasks. We propose a simple yet effective approach to convert va…
SentenceSentence SimilaritySTS