Online Distilling from Checkpoints for Neural Machine Translation
Current predominant neural machine translation (NMT) models often have a deep structure with large amounts of parameters, making these models hard to train and easily suffering from over-fitting. A common practice is to utilize a validation set to evaluate the training process and select the best checkpoint. Average and ensemble techniques on checkpoints can lead to further performance improvement. However, as these methods do not affect the training process, the system performance is restricted to the checkpoints generated in original training procedure. In contrast, we propose an online knowledge distillation method. Our method on-the-fly generates a teacher model from checkpoints, guiding the training process to obtain better performance. Experiments on several datasets and language pairs show steady improvement over a strong self-attention-based baseline system. We also provide analysis on data-limited setting against over-fitting. Furthermore, our method leads to an improvement in a machine reading experiment as well.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationMachine TranslationNMTReading ComprehensionTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Pre-trained Summarization Distillation
Recent state-of-the-art approaches to summarization utilize large pre-trained Transformer models. Distilling these models to smaller student models has become critically important for practical use; however there are man…
Knowledge DistillationMachine TranslationTranslationGrowing Together: Modeling Human Language Learning With n-Best Multi-Checkpoint Machine Translation
We describe our submission to the 2020 Duolingo Shared Task on Simultaneous Translation And Paraphrase for Language Education (STAPLE) (Mayhew et al., 2020). We view MT models at various training stages (i.e., checkpoint…
Machine TranslationTranslationRevisiting Checkpoint Averaging for Neural Machine Translation
Checkpoint averaging is a simple and effective method to boost the performance of converged neural machine translation models. The calculation is cheap to perform and the fact that the translation improvement almost come…
Machine TranslationTranslationLeveraging Pre-trained Checkpoints for Sequence Generation Tasks
Unsupervised pre-training of large neural models has recently revolutionized Natural Language Processing. By warm-starting from the publicly released checkpoints, NLP practitioners have pushed the state-of-the-art on mul…
DecoderMachine TranslationNatural Language UnderstandingSentence+5Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translation
We explore best practices for training small, memory efficient machine translation models with sequence-level knowledge distillation in the domain adaptation setting. While both domain adaptation and knowledge distillati…
Domain AdaptationKnowledge DistillationMachine TranslationTranslation