Transformer-Based Direct Hidden Markov Model for Machine Translation
The neural hidden Markov model has been proposed as an alternative to attention mechanism in machine translation with recurrent neural networks. However, since the introduction of the transformer models, its performance has been surpassed. This work proposes to introduce the concept of the hidden Markov model to the transformer architecture, which outperforms the transformer baseline. Interestingly, we find that the zero-order model already provides promising performance, giving it an edge compared to a model with first-order dependency, which performs similarly but is significantly slower in training and decoding.
Code (0)
등록된 구현이 없습니다.
Tasks
Machine TranslationTranslationSimilar Papers 제목 키워드 기반
Hidden Markov Transformer for Simultaneous Machine Translation
Simultaneous machine translation (SiMT) outputs the target sequence while receiving the source sequence, and hence learning when to start translating each target token is the core challenge for SiMT task. However, it is …
Machine TranslationTranslationNeural Hidden Markov Model for Machine Translation
Attention-based neural machine translation (NMT) models selectively focus on specific source positions to produce a translation, which brings significant improvements over pure encoder-decoder sequence-to-sequence models…
DecoderMachine TranslationmodelNMT+1Revisiting the Markov Property for Machine Translation
In this paper, we re-examine the Markov property in the context of neural machine translation. We design a Markov Autoregressive Transformer~(MAT) and undertake a comprehensive assessment of its performance across four W…
Machine TranslationTranslationDirected Acyclic Transformer for Non-Autoregressive Machine Translation
Non-autoregressive Transformers (NATs) significantly reduce the decoding latency by generating all tokens in parallel. However, such independent predictions prevent NATs from capturing the dependencies between the tokens…
Knowledge DistillationMachine TranslationTranslationAn Intrinsic Nearest Neighbor Analysis of Neural Machine Translation Architectures
Earlier approaches indirectly studied the information captured by the hidden states of recurrent and non-recurrent neural machine translation models by feeding them into different classifiers. In this paper, we look at t…
Machine TranslationTranslationWord Embeddings