On the Sparsity of Neural Machine Translation Models
Modern neural machine translation (NMT) models employ a large number of parameters, which leads to serious over-parameterization and typically causes the underutilization of computational resources. In response to this problem, we empirically investigate whether the redundant parameters can be reused to achieve better performance. Experiments and analyses are systematically conducted on different datasets and NMT architectures. We show that: 1) the pruned parameters can be rejuvenated to improve the baseline model by up to +0.8 BLEU points; 2) the rejuvenated parameters are reallocated to enhance the ability of modeling low-level lexical information.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationNMTTranslationSimilar Papers 제목 키워드 기반
Reducing the Impact of Data Sparsity in Statistical Machine Translation
Three-phase training to address data sparsity in Neural Machine Translation
Target-Side Augmentation for Document-Level Machine Translation
Document-level machine translation faces the challenge of data sparsity due to its long input length and a small amount of training data, increasing the risk of learning spurious patterns. To address this challenge, we p…
Data AugmentationDocument Level Machine TranslationMachine TranslationTranslationSemantic Neural Machine Translation using AMR
It is intuitive that semantic representations can be useful for machine translation, mainly because they can help in enforcing meaning preservation and handling data sparsity (many sentences correspond to one meaning) of…
Abstract Meaning RepresentationMachine TranslationNMTTranslationSimulated Multiple Reference Training Improves Low-Resource Machine Translation
Many valid translations exist for a given sentence, yet machine translation (MT) is trained with a single reference translation, exacerbating data sparsity in low-resource settings. We introduce Simulated Multiple Refere…
Machine TranslationSentenceTranslationvalid