Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation
Despite some empirical success at correcting exposure bias in machine translation, scheduled sampling algorithms suffer from a major drawback: they incorrectly assume that words in the reference translations and in sampled sequences are aligned at each time step. Our new differentiable sampling algorithm addresses this issue by optimizing the probability that the reference can be aligned with the sampled output, based on a soft alignment predicted by the model itself. As a result, the output distribution at each time step is evaluated with respect to the whole predicted sequence. Experiments on IWSLT translation tasks show that our approach improves BLEU compared to maximum likelihood and scheduled sampling baselines. In addition, our approach is simpler to train with no need for sampling schedule and yields models that achieve larger improvements with smaller beam sizes.
Code (1)
Tasks
Machine TranslationTranslationSimilar Papers 제목 키워드 기반
Learning What's Easy: Fully Differentiable Neural Easy-First Taggers
We introduce a novel neural easy-first decoder that learns to solve sequence tagging tasks in a flexible order. In contrast to previous easy-first decoders, our models are end-to-end differentiable. The decoder iterative…
DecoderImitation LearningNamed Entity Recognition (NER)Part-Of-Speech TaggingDiscourse Context Primes Hindi Word Order
Hindi has a flexible word order, yet certain word orders are consistently preferred over others. A number of factors are known to influence Hindi word order preferences in isolation, including information structure and s…
SentenceLanguage Models as an Alternative Evaluator of Word Order Hypotheses: A Case Study in Japanese
We examine a methodology using neural language models (LMs) for analyzing the word order of language. This LM-based method has the potential to overcome the difficulties existing methods face, such as the propagation of …
validDifferentiable DAG Sampling
We propose a new differentiable probabilistic model over DAGs (DP-DAG). DP-DAG allows fast and differentiable DAG sampling suited to continuous optimization. To this end, DP-DAG samples a DAG by successively (1) sampling…
validVariational InferenceAnchored Langevin Algorithms
Standard first-order Langevin algorithms such as the unadjusted Langevin algorithm (ULA) are obtained by discretizing the Langevin diffusion and are widely used for sampling in machine learning because they scale to high…