Combining Sequence Distillation and Transfer Learning for Efficient Low-Resource Neural Machine Translation Models
In neural machine translation (NMT), sequence distillation (SD) through creation of distilled corpora leads to efficient (compact and fast) models. However, its effectiveness in extremely low-resource (ELR) settings has not been well-studied. On the other hand, transfer learning (TL) by leveraging larger helping corpora greatly improves translation quality in general. This paper investigates a combination of SD and TL for training efficient NMT models for ELR settings, where we utilize TL with helping corpora twice: once for distilling the ELR corpora and then during compact model training. We experimented with two ELR settings: Vietnamese–English and Hindi–English from the Asian Language Treebank dataset with 18k training sentence pairs. Using the compact models with 40% smaller parameters trained on the distilled ELR corpora, greedy search achieved 3.6 BLEU points improvement in average while reducing 40% of decoding time. We also confirmed that using both the distilled ELR and helping corpora in the second round of TL further improves translation quality. Our work highlights the importance of stage-wise application of SD and TL for efficient NMT modeling for ELR settings.
Code (0)
등록된 구현이 없습니다.
Tasks
Low Resource Neural Machine TranslationLow-Resource Neural Machine TranslationMachine TranslationNMTSentenceTransfer LearningTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation
Scarcity of parallel sentence-pairs poses a significant hurdle for training high-quality Neural Machine Translation (NMT) models in bilingually low-resource scenarios. A standard approach is transfer learning, which invo…
Knowledge DistillationLow Resource Neural Machine TranslationLow-Resource Neural Machine TranslationMachine Translation+4Language Graph Distillation for Low-Resource Machine Translation
Neural machine translation on low-resource language is challenging due to the lack of bilingual sentence pairs. Previous works usually solve the low-resource translation problem with knowledge transfer in a multilingual …
Knowledge DistillationMachine TranslationSentenceTransfer Learning+1On Knowledge Distillation for Direct Speech Translation
Direct speech translation (ST) has shown to be a complex task requiring knowledge transfer from its sub-tasks: automatic speech recognition (ASR) and machine translation (MT). For MT, one of the most promising techniques…
Automatic Speech RecognitionAutomatic Speech Recognition (ASR)Knowledge DistillationMachine Translation+4Space-Time Distillation for Video Super-Resolution
Compact video super-resolution (VSR) networks can be easily deployed on resource-limited devices, e.g., smart-phones and wearable devices, but have considerable performance gaps compared with complicated VSR networks…
Knowledge DistillationSuper-ResolutionVideo Super-ResolutionDistillation from Heterogeneous Models for Top-K Recommendation
Recent recommender systems have shown remarkable performance by using an ensemble of heterogeneous models. However, it is exceedingly costly because it requires resources and inference latency proportional to the number …
Knowledge DistillationRecommendation SystemsTransfer Learning