Neural Discrete Representation Learning
Learning useful representations without supervision remains a key challenge in machine learning. In this paper, we propose a simple yet powerful generative model that learns such discrete representations. Our model, the Vector Quantised-Variational AutoEncoder (VQ-VAE), differs from VAEs in two key ways: the encoder network outputs discrete, rather than continuous, codes; and the prior is learnt rather than static. In order to learn a discrete latent representation, we incorporate ideas from vector quantisation (VQ). Using the VQ method allows the model to circumvent issues of "posterior collapse" -- where the latents are ignored when they are paired with a powerful autoregressive decoder -- typically observed in the VAE framework. Pairing these representations with an autoregressive prior, the model can generate high quality images, videos, and speech as well as doing high quality speaker conversion and unsupervised learning of phonemes, providing further evidence of the utility of the learnt representations.
Code (50)
Tasks
DecoderRepresentation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Autoregressive Co-Training for Learning Discrete Speech Representations
While several self-supervised approaches for learning discrete speech representation have been proposed, it is unclear how these seemingly similar approaches relate to each other. In this paper, we consider a generative …
QuantizationCubic Discrete Diffusion: Discrete Visual Generation on High-Dimensional Representation Tokens
Visual generation with discrete tokens has gained significant attention as it enables a unified token prediction paradigm shared with language models, promising seamless multimodal architectures. However, current discret…
Offline RL with Discrete Proxy Representations for Generalizability in POMDPs
Offline Reinforcement Learning (RL) has demonstrated promising results in various applications by learning policies from previously collected datasets, reducing the need for online exploration and interactions. However, …
Is Information in the Brain Represented in Continuous or Discrete Form?
The question of continuous-versus-discrete information representation in the brain is a fundamental yet unresolved question. Historically, most analyses assume a continuous representation without considering the discrete…
FormLearning Interpretable and Discrete Representations with Adversarial Training for Unsupervised Text Classification
Learning continuous representations from unlabeled textual data has been increasingly studied for benefiting semi-supervised learning. Although it is relatively easier to interpret discrete representations, due to the di…
General Classificationtext-classificationText ClassificationUnsupervised Text Classification