Self-supervised and Supervised Joint Training for Resource-rich Machine Translation
Self-supervised pre-training of text representations has been successfully applied to low-resource Neural Machine Translation (NMT). However, it usually fails to achieve notable gains on resource-rich NMT. In this paper, we propose a joint training approach, $F_2$-XEnDec, to combine self-supervised and supervised learning to optimize NMT models. To exploit complementary self-supervised signals for supervised learning, NMT models are trained on examples that are interbred from monolingual and parallel sentences through a new process called crossover encoder-decoder. Experiments on two resource-rich translation benchmarks, WMT'14 English-German and WMT'14 English-French, demonstrate that our approach achieves substantial improvements over several strong baseline methods and obtains a new state of the art of 46.19 BLEU on English-French when incorporating back translation. Results also show that our approach is capable of improving model robustness to input perturbations such as code-switching noise which frequently appears on social media.
Code (0)
등록된 구현이 없습니다.
Tasks
DecoderLow Resource Neural Machine TranslationLow-Resource Neural Machine TranslationMachine TranslationNMTTranslationSimilar Papers 제목 키워드 기반
Joint Unsupervised and Supervised Training for Multilingual ASR
Self-supervised training has shown promising gains in pretraining models and facilitating the downstream finetuning for speech recognition, like multilingual ASR. Most existing methods adopt a 2-stage scheme where the se…
Language ModelingLanguage ModellingMasked Language Modelingspeech-recognition+2Unsupervised Representation Learning for Binary Networks by Joint Classifier Learning
Self-supervised learning is a promising unsupervised learning framework that has achieved success with large floating point networks. But such networks are not readily deployable to edge devices. To accelerate deployment…
Representation LearningSelf-Supervised LearningJointly Learning Visual and Auditory Speech Representations from Raw Data
We present RAVEn, a self-supervised multi-modal approach to jointly learn visual and auditory speech representations. Our pre-training objective involves encoding masked inputs, and then predicting contextualised targets…
Audio-Visual Speech RecognitionLipreadingspeech-recognitionSpeech Recognition+1Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning
Self-supervised learning is an efficient pre-training method for medical image analysis. However, current research is mostly confined to specific-modality data pre-training, consuming considerable time and resources with…
Continual LearningContinual Self-Supervised LearningMedical Image AnalysisRepresentation Learning+1Improving label efficiency through multi-task learning on auditory data
Collecting high-quality, large scale datasets typically requires significant resources. The aim of the present work is to improve the label efficiency of large neural networks operating on audio data through multitask le…
Data AugmentationMulti-Task LearningSelf-Supervised Learning