Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation
Back-translation is a critical component of Unsupervised Neural Machine Translation (UNMT), which generates pseudo parallel data from target monolingual data. A UNMT model is trained on the pseudo parallel data with $\text{\bf translated source}$, and translates $\text{\bf natural source}$ sentences in inference. The source discrepancy between training and inference hinders the translation performance of UNMT models. By carefully designing experiments, we identify two representative characteristics of the data gap in source: (1) $\text{\textit{style gap}}$ (i.e., translated vs. natural text style) that leads to poor generalization capability; (2) $\text{\textit{content gap}}$ that induces the model to produce hallucination content biased towards the target language. To narrow the data gap, we propose an online self-training approach, which simultaneously uses the pseudo parallel data $\{$natural source, translated target$\}$ to mimic the inference scenario. Experimental results on several widely-used language pairs show that our approach outperforms two strong baselines (XLM and MASS) by remedying the style and content gaps.
Code (0)
등록된 구현이 없습니다.
Tasks
HallucinationMachine TranslationTranslationSimilar Papers 제목 키워드 기반
Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation
Back-translation is a critical component of Unsupervised Neural Machine Translation (UNMT), which generates pseudo parallel data from target monolingual data. A UNMT model is trained on the pseudo parallel data with tran…
HallucinationMachine TranslationTranslationBridging the Gap between Language Model and Reading Comprehension: Unsupervised MRC via Self-Supervision
Despite recent success in machine reading comprehension (MRC), learning high-quality MRC models still requires large-scale labeled training data, even using strong pre-trained language models (PLMs). The pre-training tas…
Language ModelingLanguage ModellingMachine Reading ComprehensionQuestion Answering+4Probing for Bridging Inference in Transformer Language Models
We probe pre-trained transformer language models for bridging inference. We first investigate individual attention heads in BERT and observe that attention heads at higher layers prominently focus on bridging relations i…
Bridging Anaphora ResolutionCloze TestCLISC: Bridging clip and sam by enhanced cam for unsupervised brain tumor segmentation
Brain tumor segmentation is important for diagnosis of the tumor, and current deep-learning methods rely on a large set of annotated images for training, with high annotation costs. Unsupervised segmentation is promising…
Brain Tumor SegmentationData AugmentationSegmentationSelf-Learning+1Scheduled Sampling Based on Decoding Steps for Neural Machine Translation
Scheduled sampling is widely used to mitigate the exposure bias problem for neural machine translation. Its core motivation is to simulate the inference scene during training by replacing ground-truth tokens with predict…
Machine TranslationText SummarizationTranslation