Non-Autoregressive Translation by Learning Target Categorical Codes
Non-autoregressive Transformer is a promising text generation model. However, current non-autoregressive models still fall behind their autoregressive counterparts in translation quality. We attribute this accuracy gap to the lack of dependency modeling among decoder inputs. In this paper, we propose CNAT, which learns implicitly categorical codes as latent variables into the non-autoregressive decoding. The interaction among these categorical codes remedies the missing dependencies and improves the model capacity. Experiment results show that our model achieves comparable or better performance in machine translation tasks, compared with several strong baselines.
Code (1)
Tasks
AttributeDecoderMachine TranslationText GenerationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Syntactically Supervised Transformers for Faster Neural Machine Translation
Standard decoders for neural machine translation autoregressively generate a single target token per time step, which slows inference especially for long outputs. While architectural advances such as the Transformer full…
DecoderMachine TranslationTranslationMR-P: A Parallel Decoding Algorithm for Iterative Refinement Non-Autoregressive Translation
Non-autoregressive translation (NAT) predicts all the target tokens in parallel and significantly speeds up the inference process. The Conditional Masked Language Model (CMLM) is a strong baseline of NAT. It decodes with…
Language ModelingLanguage ModellingTranslationIntegrating Categorical Semantics into Unsupervised Domain Translation
While unsupervised domain translation (UDT) has seen a lot of success recently, we argue that mediating its translation via categorical semantic features could broaden its applicability. In particular, we demonstrate tha…
ObjectTranslationImproving Autoregressive NMT with Non-Autoregressive Model
Autoregressive neural machine translation (NMT) models are often used to teach non-autoregressive models via knowledge distillation. However, there are few studies on improving the quality of autoregressive translation (…
Decoderde-enKnowledge DistillationMachine Translation+3Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation
Non-Autoregressive Transformer (NAT) aims to accelerate the Transformer model through discarding the autoregressive mechanism and generating target words independently, which fails to exploit the target sequential inform…
DecoderMachine TranslationSentenceTranslation