Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation
Self-supervised neural machine translation (SSNMT) jointly learns to identify and select suitable training data from comparable (rather than parallel) corpora and to translate, in a way that the two tasks support each other in a virtuous circle. In this study, we provide an in-depth analysis of the sampling choices the SSNMT model makes during training. We show how, without it having been told to do so, the model self-selects samples of increasing (i) complexity and (ii) task-relevance in combination with (iii) performing a denoising curriculum. We observe that the dynamics of the mutual-supervision signals of both system internal representation types are vital for the extraction and translation performance. We show that in terms of the Gunning-Fog Readability index, SSNMT starts extracting and learning from Wikipedia data suitable for high school students and quickly moves towards content suitable for first year undergraduate students.
Code (0)
등록된 구현이 없습니다.
Tasks
DenoisingMachine TranslationTranslationSimilar Papers 제목 키워드 기반
Self-Induced Curriculum Learning in Neural Machine Translation
Self-supervised neural machine translation (SS-NMT) learns how to extract/select suitable training data from comparable (rather than parallel) corpora and how to translate, in a way that the two tasks support each other …
DenoisingMachine TranslationNMTTranslationDARC: Decoupled Asymmetric Reasoning Curriculum for LLM Evolution
Self-play with large language models has emerged as a promising paradigm for achieving self-improving artificial intelligence. However, existing self-play frameworks often suffer from optimization instability, due to (i)…
Curriculum learning for self-supervised speaker verification
The goal of this paper is to train effective self-supervised speaker representations without identity labels. We propose two curriculum learning strategies within a self-supervised learning framework. The first strategy …
Self-Supervised LearningSpeaker RecognitionSpeaker VerificationPointSmile: Point Self-supervised Learning via Curriculum Mutual Information
Self-supervised learning is attracting wide attention in point cloud processing. However, it is still not well-solved to gain discriminative and transferable features of point clouds for efficient training on downstream …
Data AugmentationSelf-Supervised LearningDiversity Induced Environment Design via Self-Play
Recent work on designing an appropriate distribution of environments has shown promise for training effective generally capable agents. Its success is partly because of a form of adaptive curriculum learning that generat…
Diversity