Improving Autoregressive NMT with Non-Autoregressive Model
Autoregressive neural machine translation (NMT) models are often used to teach non-autoregressive models via knowledge distillation. However, there are few studies on improving the quality of autoregressive translation (AT) using non-autoregressive translation (NAT). In this work, we propose a novel Encoder-NAD-AD framework for NMT, aiming at boosting AT with global information produced by NAT model. Specifically, under the semantic guidance of source-side context captured by the encoder, the non-autoregressive decoder (NAD) first learns to generate target-side hidden state sequence in parallel. Then the autoregressive decoder (AD) performs translation from left to right, conditioned on source-side and target-side hidden states. Since AD has global information generated by low-latency NAD, it is more likely to produce a better translation with less time delay. Experiments on WMT14 En-De, WMT16 En-Ro, and IWSLT14 De-En translation tasks demonstrate that our framework achieves significant improvements with only 8{\%} speed degeneration over the autoregressive NMT.
Code (0)
등록된 구현이 없습니다.
Tasks
Decoderde-enKnowledge DistillationMachine TranslationmodelNMTTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation
We propose to train a non-autoregressive machine translation model to minimize the energy defined by a pretrained autoregressive model. In particular, we view our non-autoregressive translation system as an inference net…
de-enMachine TranslationTranslationHAINAN: Fast and Accurate Transducer for Hybrid-Autoregressive ASR
We present Hybrid-Autoregressive INference TrANsducers (HAINAN), a novel architecture for speech recognition that extends the Token-and-Duration Transducer (TDT) model. Trained with randomly masked predictor network outp…
speech-recognitionSpeech RecognitionDeep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
Much recent effort has been invested in non-autoregressive neural machine translation, which appears to be an efficient alternative to state-of-the-art autoregressive machine translation on modern GPUs. In contrast to th…
DecoderKnowledge DistillationMachine TranslationTranslationMasked Autoregressive Flow for Density Estimation
Autoregressive models are among the best performing neural density estimators. We describe an approach for increasing the flexibility of an autoregressive model, based on modelling the random numbers that the model uses …
Density EstimationImplicit Stacked Autoregressive Model for Video Prediction
Future frame prediction has been approached through two primary methods: autoregressive and non-autoregressive. Autoregressive methods rely on the Markov assumption and can achieve high accuracy in the early stages of pr…
modelPredictionVideo PredictionWeather Forecasting