Pseudo-Label Training and Model Inertia in Neural Machine Translation
Like many other machine learning applications, neural machine translation (NMT) benefits from over-parameterized deep neural models. However, these models have been observed to be brittle: NMT model predictions are sensitive to small input changes and can show significant variation across re-training or incremental model updates. This work studies a frequently used method in NMT, pseudo-label training (PLT), which is common to the related techniques of forward-translation (or self-training) and sequence-level knowledge distillation. While the effect of PLT on quality is well-documented, we highlight a lesser-known effect: PLT can enhance a model's stability to model updates and input perturbations, a set of properties we call model inertia. We study inertia effects under different training settings and we identify distribution simplification as a mechanism behind the observed results.
Code (0)
등록된 구현이 없습니다.
Tasks
Knowledge DistillationMachine TranslationNMTPseudo LabelTranslationSimilar Papers 제목 키워드 기반
Air-Writing Translater: A Novel Unsupervised Domain Adaptation Method for Inertia-Trajectory Translation of In-air Handwriting
As a new way of human-computer interaction, inertial sensor based in-air handwriting can provide a natural and unconstrained interaction to express more complex and richer information in 3D space. However, most of the ex…
Domain AdaptationTranslationUnsupervised Domain AdaptationSelf-Training for End-to-End Speech Translation
One of the main challenges for end-to-end speech translation is data scarcity. We leverage pseudo-labels generated from unlabeled audio by a cascade and an end-to-end speech translation model. This provides 8.3 and 5.7 B…
speech-recognitionSpeech RecognitionTranslationContrastive Clustering to Mine Pseudo Parallel Data for Unsupervised Translation
Modern unsupervised machine translation systems mostly train their models by generating synthetic parallel training data from large unlabeled monolingual corpora of different languages through various means, such as iter…
ClusteringMachine TranslationTranslationUnsupervised Machine TranslationJoint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data
Self-training has been shown to be helpful in addressing data scarcity for many domains, including vision, speech, and language. Specifically, self-training, or pseudo-labeling, labels unsupervised data and adds that to …
Data AugmentationPseudo LabelPseudo Label FilteringTranslationAlleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimation
Quality Estimation (QE) models evaluate the quality of machine translations without reference translations, serving as the reward models for the translation task. Due to the data scarcity, synthetic data generation has e…
Machine TranslationSynthetic Data GenerationTranslation