paper-with-me

홈 › Papers

Re-examination of the Role of Latent Variables in Sequence Modeling

2019-02-04 · NeurIPS 2019 12 · Zihang Dai, Guokun Lai, Yiming Yang, Shinjae Yoo

With latent variables, stochastic recurrent models have achieved state-of-the-art performance in modeling sound-wave sequence. However, opposite results are also observed in other domains, where standard recurrent networks often outperform stochastic models. To better understand this discrepancy, we re-examine the roles of latent variables in stochastic recurrent models for speech density estimation. Our analysis reveals that under the restriction of fully factorized output distribution in previous evaluations, the stochastic models were implicitly leveraging intra-step correlation but the standard recurrent baselines were prohibited to do so, resulting in an unfair comparison. To correct the unfairness, we remove such restriction in our re-examination, where all the models can explicitly leverage intra-step correlation with an auto-regressive structure. Over a diverse set of sequential data, including human speech, MIDI music, handwriting trajectory and frame-permuted speech, our results show that stochastic recurrent models fail to exhibit any practical advantage despite the claimed theoretical superiority. In contrast, standard recurrent models equipped with an auto-regressive output distribution consistently perform better, significantly advancing the state-of-the-art results on three speech datasets.

📄 PDF Abstract BibTeX arXiv:1902.01388

Code (1)

zihangdai/reexamine-srnn 공식 구현 pytorch

Tasks

Density Estimation

Similar Papers 제목 키워드 기반

Bandit Learning with Implicit Feedback

2018-12-01 · NeurIPS 2018 12 · Yi Qi, Qingyun Wu, Hongning Wang, Jie Tang 외

Implicit feedback, such as user clicks, although abundant in online information service systems, does not provide substantial evidence on users' evaluation of system's output. Without proper modeling, such incomplete sup…

Bayesian InferenceThompson Sampling

Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles

2022-06-22 · Ghazi Felhi, Joseph Le Roux, Djamé Seddah

Linking neural representations to linguistic factors is crucial in order to build and analyze NLP models interpretable by humans. Among these factors, syntactic roles (e.g. subjects, direct objects,$\dots$) and their rea…

DecoderDisentanglementMachine TranslationSentence

Variational Connectionist Temporal Classification for Order-Preserving Sequence Modeling

2023-09-21 · Zheng Nan, Ting Dang, Vidhyasaharan Sethu, Beena Ahmed

Connectionist temporal classification (CTC) is commonly adopted for sequence modeling tasks like speech recognition, where it is necessary to preserve order between the input and target sequences. However, CTC is only ap…

Classificationspeech-recognitionSpeech Recognition

Variational Hyper RNN for Sequence Modeling

2020-02-24 · Ruizhi Deng, Yanshuai Cao, Bo Chang, Leonid Sigal 외

In this work, we propose a novel probabilistic sequence model that excels at capturing high variability in time series data, both across sequences and within an individual sequence. Our method uses temporal latent variab…

DecoderTime SeriesTime Series Analysis

Multi-level Convolutional Autoencoder Networks for Parametric Prediction of Spatio-temporal Dynamics

2019-12-23 · Jiayang Xu, Karthik Duraisamy

A data-driven framework is proposed towards the end of predictive modeling of complex spatio-temporal dynamics, leveraging nested non-linear manifolds. Three levels of neural networks are used, with the goal of predictin…

Temporal Sequences