On the Interaction of Regularization Factors in Low-resource Neural Machine Translation
We explore the roles and interactions of the hyper-parameters governing regularization, and propose a range of values applicable to low-resource neural machine translation. We demonstrate that default or recommended values for high-resource settings are not optimal for low-resource ones, and that more aggressive regularization is needed when resources are scarce, in proportion to their scarcity. We explain our observations by the generalization abilities of sharp vs. flat basins in the loss landscape of a neural network. Results for four regularization factors corroborate our claim: batch size, learning rate, dropout rate, and gradient clipping. Moreover, we show that optimal results are obtained when using several of these factors, and that our findings generalize across datasets of different sizes and languages.
Code (1)
Tasks
Low Resource Neural Machine TranslationLow-Resource Neural Machine TranslationMachine TranslationTranslationSimilar Papers 제목 키워드 기반
A Systematic Study Reveals Unexpected Interactions in Pre-Trained Neural Machine Translation
A significant challenge in developing translation systems for the world’s ∼7,000 languages is that very few have sufficient data for state-of-the-art techniques. Transfer learning is a promising direction for low-resourc…
Low Resource Neural Machine TranslationLow-Resource Neural Machine TranslationLow Resource NMTMachine Translation+3Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation
Sparsely gated Mixture of Experts (MoE) models have been shown to be a compute-efficient method to scale model capacity for multilingual machine translation. However, for low-resource tasks, MoE models severely over-fit.…
Machine TranslationMixture-of-ExpertsTranslationSingle Model Ensemble for Subword Regularized Models in Low-Resource Machine Translation
Subword regularizations use multiple subword segmentations during training to improve the robustness of neural machine translation models. In previous subword regularizations, we use multiple segmentations in the trainin…
Machine TranslationSegmentationTranslationLimited-Resource Adapters Are Regularizers, Not Linguists
Cross-lingual transfer from related high-resource languages is a well-established strategy to enhance low-resource language technologies. Prior work has shown that adapters show promise for, e.g., improving low-resource …
Cross-Lingual TransferMachine TranslationReWE: Regressing Word Embeddings for Regularization of Neural Machine Translation Systems
Regularization of neural machine translation is still a significant problem, especially in low-resource settings. To mollify this problem, we propose regressing word embeddings (ReWE) as a new regularization technique in…
Machine TranslationTranslationWord Embeddings