Improving Variational Autoencoder for Text Modelling with Timestep-Wise Regularisation
The Variational Autoencoder (VAE) is a popular and powerful model applied to text modelling to generate diverse sentences. However, an issue known as posterior collapse (or KL loss vanishing) happens when the VAE is used in text modelling, where the approximate posterior collapses to the prior, and the model will totally ignore the latent variables and be degraded to a plain language model during text generation. Such an issue is particularly prevalent when RNN-based VAE models are employed for text modelling. In this paper, we propose a simple, generic architecture called Timestep-Wise Regularisation VAE (TWR-VAE), which can effectively avoid posterior collapse and can be applied to any RNN-based VAE models. The effectiveness and versatility of our model are demonstrated in different tasks, including language modelling and dialogue response generation.
Code (1)
Tasks
Language ModelingLanguage ModellingResponse GenerationText GenerationMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
A Stable Variational Autoencoder for Text Modelling
Variational Autoencoder (VAE) is a powerful method for learning representations of high-dimensional data. However, VAEs can suffer from an issue known as latent variable collapse (or KL loss vanishing), where the posteri…
Variational Autoencoders for Sparse and Overdispersed Discrete Data
Many applications, such as text modelling, high-throughput sequencing, and recommender systems, require analysing sparse, high-dimensional, and overdispersed discrete (count-valued or binary) data. Although probabilistic…
Collaborative FilteringMulti-Label LearningRecommendation SystemsVariational Autoencoders Without the Variation
Variational autoencdoers (VAE) are a popular approach to generative modelling. However, exploiting the capabilities of VAEs in practice can be difficult. Recent work on regularised and entropic autoencoders have begun to…
Image GenerationLocal Information Opponent Modelling Using Variational Autoencoders
Modelling the behaviours of other agents (opponents) is essential for understanding how agents interact and making effective decisions. Existing methods for opponent modelling commonly assume knowledge of the local obser…
Deep Reinforcement LearningDiffusion Priors In Variational Autoencoders
Among likelihood-based approaches for deep generative modelling, variational autoencoders (VAEs) offer scalable amortized posterior inference and fast sampling. However, VAEs are also more and more outperformed by compet…
Denoising