Improved Variational Neural Machine Translation by Promoting Mutual Information
Posterior collapse plagues VAEs for text, especially for conditional text generation with strong autoregressive decoders. In this work, we address this problem in variational neural machine translation by explicitly promoting mutual information between the latent variables and the data. Our model extends the conditional variational autoencoder (CVAE) with two new ingredients: first, we propose a modified evidence lower bound (ELBO) objective which explicitly promotes mutual information; second, we regularize the probabilities of the decoder by mixing an auxiliary factorized distribution which is directly predicted by the latent variables. We present empirical results on the Transformer architecture and show the proposed model effectively addressed posterior collapse: latent variables are no longer ignored in the presence of powerful decoder. As a result, the proposed model yields improved translation quality while demonstrating superior performance in terms of data efficiency and robustness.
Code (0)
등록된 구현이 없습니다.
Tasks
Conditional Text GenerationDecoderMachine TranslationText GenerationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Addressing Posterior Collapse with Mutual Information for Improved Variational Neural Machine Translation
This paper proposes a simple and effective approach to address the problem of posterior collapse in conditional variational autoencoders (CVAEs). It thus improves performance of machine translation models that use noisy …
de-enMachine TranslationNMTTranslationCut the noise: Mutually reinforcing reordering and alignments for improved machine translation
Diversity-Promoting Bayesian Learning of Latent Variable Models
To address three important issues involved in latent variable models (LVMs), including capturing infrequent patterns, achieving small-sized but expressive models and alleviating overfitting, several studies have been dev…
DiversityMixture-of-ExpertsVariational InferenceVariational Neural Machine Translation
Models of neural machine translation are often from a discriminative family of encoderdecoders that learn a conditional distribution of a target sentence given a source sentence. In this paper, we propose a variational m…
DecoderMachine TranslationSentenceTranslationOn Systematic Style Differences between Unsupervised and Supervised MT and an Application for High-Resource Machine Translation
Modern unsupervised machine translation (MT) systems reach reasonable translation quality under clean and controlled data conditions. As the performance gap between supervised and unsupervised MT narrows, it is interesti…
Machine TranslationTranslationUnsupervised Machine Translation