Variational recurrent models for representation learning
We study the problem of learning representations of sequence data. Recent work has built on variational autoencoders to develop variational recurrent models for generation. Our main goal is not generation but rather representation learning for downstream prediction tasks. Existing variational recurrent models typically use stochastic recurrent connections to model the dependence among neighboring latent variables, while generation assumes independence of generated data per time step given the latent sequence. In contrast, our models assume independence among all latent variables given non-stochastic hidden states, which speeds up inference, while assuming dependence of observations at each time step on all latent variables, which improves representation quality. In addition, we propose and study extensions for improving downstream performance, including hierarchical auxiliary latent variables and prior updating during training. Experiments show improved performance on several speech and language tasks with different levels of supervision, as well as in a multi-view learning setting.
Code (0)
등록된 구현이 없습니다.
Tasks
MULTI-VIEW LEARNINGRepresentation LearningSimilar Papers 제목 키워드 기반
Note on Equivalence Between Recurrent Neural Network Time Series Models and Variational Bayesian Models
We observe that the standard log likelihood training objective for a Recurrent Neural Network (RNN) model of time series data is equivalent to a variational Bayesian training objective, given the proper choice of generat…
Time SeriesTime Series AnalysisVariational Graph Recurrent Neural Networks
Representation learning over graph structured data has been mostly studied in static graph settings while efforts for modeling dynamic graphs are still scant. In this paper, we develop a novel hierarchical variational mo…
AttributeDynamic Link PredictionLink PredictionRepresentation Learning+1Variational Recurrent Auto-Encoders
In this paper we propose a model that combines the strengths of RNNs and SGVB: the Variational Recurrent Auto-Encoder (VRAE). Such a model can be used for efficient, large scale unsupervised learning on time series data,…
Time SeriesTime Series AnalysisGraph Generation with Variational Recurrent Neural Network
Generating graph structures is a challenging problem due to the diverse representations and complex dependencies among nodes. In this paper, we introduce Graph Variational Recurrent Neural Network (GraphVRNN), a probabil…
AttributeGraph GenerationGraph structure learningVariational Inference-Based Dropout in Recurrent Neural Networks for Slot Filling in Spoken Language Understanding
This paper proposes to generalize the variational recurrent neural network (RNN) with variational inference (VI)-based dropout regularization employed for the long short-term memory (LSTM) cells to more advanced RNN arch…
slot-fillingSlot FillingSpoken Language UnderstandingVariational Inference