Incorporating a Local Translation Mechanism into Non-autoregressive Translation
In this work, we introduce a novel local autoregressive translation (LAT) mechanism into non-autoregressive translation (NAT) models so as to capture local dependencies among tar-get outputs. Specifically, for each target decoding position, instead of only one token, we predict a short sequence of tokens in an autoregressive way. We further design an efficient merging algorithm to align and merge the out-put pieces into one final output sequence. We integrate LAT into the conditional masked language model (CMLM; Ghazvininejad et al.,2019) and similarly adopt iterative decoding. Empirical results on five translation tasks show that compared with CMLM, our method achieves comparable or better performance with fewer decoding iterations, bringing a 2.5xspeedup. Further analysis indicates that our method reduces repeated translations and performs better at longer sentences.
Code (1)
Tasks
Language ModelingMachine TranslationPositionTARTranslationSimilar Papers 제목 키워드 기반
Improving Non-Autoregressive Neural Machine Translation via Modeling Localness
Non-autoregressive translation (NAT) models, which eliminate the sequential dependencies within the target sentence, have achieved remarkable inference speed, but suffer from inferior translation quality. Towards explori…
DecoderMachine TranslationSentenceTranslationIntegrating Translation Memories into Non-Autoregressive Machine Translation
Non-autoregressive machine translation (NAT) has recently made great progress. However, most works to date have focused on standard translation tasks, even though some edit-based NAT models, such as the Levenshtein Trans…
Knowledge DistillationMachine TranslationTranslationAutoregressive Moving-average Attention Mechanism for Time Series Forecasting
We propose an Autoregressive (AR) Moving-average (MA) attention structure that can adapt to various linear attention mechanisms, enhancing their ability to capture long-range and local temporal patterns in time series. I…
DecoderTime SeriesTime Series ForecastingInfusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism
Non-autoregressive models generate target words in a parallel way, which achieve a faster decoding speed but at the sacrifice of translation accuracy. To remedy a flawed translation by non-autoregressive models, a promis…
DecoderKnowledge DistillationTranslationIncorporating Terminology Constraints in Automatic Post-Editing
Users of machine translation (MT) may want to ensure the use of specific lexical terminologies. While there exist techniques for incorporating terminology constraints during inference for MT, current APE approaches canno…
Automatic Post-EditingData AugmentationMachine TranslationTranslation