Does Multi-Encoder Help? A Case Study on Context-Aware Neural Machine Translation
In encoder-decoder neural models, multiple encoders are in general used to represent the contextual information in addition to the individual sentence. In this paper, we investigate multi-encoder approaches in documentlevel neural machine translation (NMT). Surprisingly, we find that the context encoder does not only encode the surrounding sentences but also behaves as a noise generator. This makes us rethink the real benefits of multi-encoder in context-aware translation - some of the improvements come from robust training. We compare several methods that introduce noise and/or well-tuned dropout setup into the training of these encoders. Experimental results show that noisy training plays an important role in multi-encoder-based NMT, especially when the training data is small. Also, we establish a new state-of-the-art on IWSLT Fr-En task by careful use of noise generation and dropout methods.
Code (1)
Tasks
Decoderfr-enMachine TranslationNMTSentenceTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Case Study on Learning a Unified Encoder of Relations
Typical relation extraction models are trained on a single corpus annotated with a pre-defined relation schema. An individual corpus is often small, and the models may often be biased or overfitted to the corpus. We hypo…
Knowledge Base PopulationRelationRelation ExtractionA Case Study on Context Encoding in Multi-Encoder based Document-Level Neural Machine Translation
Recent studies have shown that the multi-encoder models are agnostic to the choice of context, and the context encoder generates noise which helps improve the models in terms of BLEU score. In this paper, we further expl…
Machine TranslationTranslationMore Context, Larger Models, or Moral Knowledge? A Systematic Study of Schwartz Value Detection in Political Texts
Detecting Schwartz values in political text is difficult because implicit cues often depend on surrounding arguments and fine-grained distinctions between neighboring values. We study when context and explicit moral know…
A Compact and Language-Sensitive Multilingual Translation Method
Multilingual neural machine translation (Multi-NMT) with one encoder-decoder model has made remarkable progress due to its simple deployment. However, this multilingual translation paradigm does not make full use of lang…
DecoderMachine TranslationNMTTranslationAn Empirical Study on Finding Spans
We present an empirical study on methods for span finding, the selection of consecutive tokens in text for some downstream tasks. We focus on approaches that can be employed in training end-to-end information extraction …
Prediction