Self-Induced Curriculum Learning in Neural Machine Translation
Self-supervised neural machine translation (SS-NMT) learns how to extract/select suitable training data from comparable (rather than parallel) corpora and how to translate, in a way that the two tasks support each other in a virtuous circle. SS-NMT has been shown to be competitive with state-of-the-art unsupervised NMT. In this study we provide an in-depth analysis of the sampling choices the SS-NMT model takes during training. We show that, without it having been told to do so, the model selects samples of increasing (i) complexity and (ii) task-relevance in combination with (iii) a denoising curriculum. We observe that the dynamics of the mutual-supervision of both system internal representation types is vital for the extraction and hence translation performance. We show that in terms of the human Gunning-Fog Readability index (GF), SS-NMT starts by extracting and learning from Wikipedia data suitable for high school (GF=10--11) and quickly moves towards content suitable for first year undergraduate students (GF=13).
Code (0)
등록된 구현이 없습니다.
Tasks
DenoisingMachine TranslationNMTTranslationSimilar Papers 제목 키워드 기반
Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation
Self-supervised neural machine translation (SSNMT) jointly learns to identify and select suitable training data from comparable (rather than parallel) corpora and to translate, in a way that the two tasks support each ot…
DenoisingMachine TranslationTranslationCompetence-based Curriculum Learning for Multilingual Machine Translation
Currently, multilingual machine translation is receiving more and more attention since it brings better performance for low resource languages (LRLs) and saves more space. However, existing multilingual machine translati…
Machine TranslationTranslationSelf-Guided Curriculum Learning for Neural Machine Translation
In the field of machine learning, the well-trained model is assumed to be able to recover the training labels, i.e. the synthetic labels predicted by the model should be as close to the ground-truth labels as possible. I…
Machine TranslationNMTSentenceTranslationAn Empirical Exploration of Curriculum Learning for Neural Machine Translation
Machine translation systems based on deep neural networks are expensive to train. Curriculum learning aims to address this issue by choosing the order in which samples are presented during training to help train better m…
Machine TranslationTranslationSelf-Paced Learning for Neural Machine Translation
Recent studies have proven that the training of neural machine translation (NMT) can be facilitated by mimicking the learning process of humans. Nevertheless, achievements of such kind of curriculum learning rely on the …
Machine TranslationNMTSentenceTranslation