Variational Transformers for Diverse Response Generation
Despite the great promise of Transformers in many sequence modeling tasks (e.g., machine translation), their deterministic nature hinders them from generalizing to high entropy tasks such as dialogue response generation. Previous work proposes to capture the variability of dialogue responses with a recurrent neural network (RNN)-based conditional variational autoencoder (CVAE). However, the autoregressive computation of the RNN limits the training efficiency. Therefore, we propose the Variational Transformer (VT), a variational self-attentive feed-forward sequence model. The VT combines the parallelizability and global receptive field of the Transformer with the variational nature of the CVAE by incorporating stochastic latent variables into Transformers. We explore two types of the VT: 1) modeling the discourse-level diversity with a global latent variable; and 2) augmenting the Transformer decoder with a sequence of fine-grained latent variables. Then, the proposed models are evaluated on three conversational datasets with both automatic metric and human evaluation. The experimental results show that our models improve standard Transformers and other baselines in terms of diversity, semantic relevance, and human judgment.
Code (2)
Tasks
DecoderDiversityMachine TranslationResponse GenerationTranslationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Towards Diverse, Relevant and Coherent Open-Domain Dialogue Generation via Hybrid Latent Variables
Conditional variational models, using either continuous or discrete latent variables, are powerful for open-domain dialogue response generation. However, previous works show that continuous latent variables tend to reduc…
Dialogue GenerationResponse GenerationImproving Contextual Coherence in Variational Personalized and Empathetic Dialogue Agents
In recent years, latent variable models, such as the Conditional Variational Auto Encoder (CVAE), have been applied to both personalized and empathetic dialogue generation. Prior work have largely focused on generating d…
Dialogue GenerationResponse GenerationKnowledge-Grounded Response Generation with Deep Attentional Latent-Variable Model
End-to-end dialogue generation has achieved promising results without using handcrafted features and attributes specific for each task and corpus. However, one of the fatal drawbacks in such approaches is that they are u…
Dialogue GenerationResponse GenerationMany Minds from One Model: Bayesian-Inspired Transformers for Population Diversity
Despite their scale and success, modern transformers are usually trained as single-minded systems: optimization produces a deterministic set of parameters, representing a single functional hypothesis about the data. Moti…
Reinforcement LearningDior-CVAE: Pre-trained Language Models and Diffusion Priors for Variational Dialog Generation
Current variational dialog models have employed pre-trained language models (PLMs) to parameterize the likelihood and posterior distributions. However, the Gaussian assumption made on the prior distribution is incompatib…
DecoderDiversityOpen-Domain DialogResponse Generation