Ensemble Distillation for Neural Machine Translation
Knowledge distillation describes a method for training a student network to perform better by learning from a stronger teacher network. Translating a sentence with an Neural Machine Translation (NMT) engine is time expensive and having a smaller model speeds up this process. We demonstrate how to transfer the translation quality of an ensemble and an oracle BLEU teacher network into a single NMT system. Further, we present translation improvements from a teacher network that has the same architecture and dimensions of the student network. As the training of the student model is still expensive, we introduce a data filtering method based on the knowledge of the teacher model that not only speeds up the training, but also leads to better translation quality. Our techniques need no code change and can be easily reproduced with any NMT architecture to speed up the decoding process.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationMachine TranslationNMTSentenceTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Knowledge Distillation based Ensemble Learning for Neural Machine Translation
Model ensemble can effectively improve the accuracy of neural machine translation, which is accompanied by the cost of large computation and memory requirements. Additionally, model ensemble cannot combine the strengths …
Ensemble LearningKnowledge DistillationMachine TranslationTranslationEBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translation
The ability of zero-shot translation emerges when we train a multilingual model with certain translation directions; the model can then directly translate in unseen directions. Alternatively, zero-shot translation can be…
Machine TranslationTranslationZero-Shot Machine TranslationThe NiuTrans Machine Translation Systems for WMT21
This paper describes NiuTrans neural machine translation systems of the WMT 2021 news translation tasks. We made submissions to 9 language directions, including English$\leftrightarrow$$\{$Chinese, Japanese, Russian, Ice…
Knowledge DistillationMachine TranslationTranslationThe Mininglamp Machine Translation System for WMT21
This paper describes Mininglamp neural machine translation systems of the WMT2021 news translation tasks. We have participated in eight directions translation tasks for news text including Chinese to/from English, Hausa …
Knowledge DistillationMachine TranslationTranslationEnsemble Distillation for Structured Prediction: Calibrated, Accurate, Fast-Choose Three
Modern neural networks do not always produce well-calibrated predictions, even when trained with a proper scoring function such as cross-entropy. In classification settings, simple methods such as isotonic regression or …
Machine Translationnamed-entity-recognitionNamed Entity RecognitionNamed Entity Recognition (NER)+2