Neural Syntactic Generative Models with Exact Marginalization
We present neural syntactic generative models with exact marginalization that support both dependency parsing and language modeling. Exact marginalization is made tractable through dynamic programming over shift-reduce parsing and minimal RNN-based feature sets. Our algorithms complement previous approaches by supporting batched training and enabling online computation of next word probabilities. For supervised dependency parsing, our model achieves a state-of-the-art result among generative approaches. We also report empirical results on unsupervised syntactic models and their role in language modeling. We find that our model formulation of latent dependencies with exact marginalization do not lead to better intrinsic language modeling performance than vanilla RNNs, and that parsing accuracy is not correlated with language modeling perplexity in stack-based models.
Code (0)
등록된 구현이 없습니다.
Tasks
Dependency ParsingLanguage ModelingLanguage ModellingTransition-Based Dependency ParsingSimilar Papers 제목 키워드 기반
Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity
Training neural network models with discrete (categorical or structured) latent variables can be computationally challenging, due to the need for marginalization over large or combinatorial sets. To circumvent this issue…
Improving NLP through Marginalization of Hidden Syntactic Structure
Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models
Many applications of generative models rely on the marginalization of their high-dimensional output probability distributions. Normalization functions that yield sparse probability distributions can make exact marginaliz…
Generative Marginalization Models
We introduce marginalization models (MAMs), a new family of generative models for high-dimensional discrete data. They offer scalable and flexible generative modeling by explicitly modeling all induced marginal distribut…
Explaining Away Syntactic Structure in Semantic Document Representations
Most generative document models act on bag-of-words input in an attempt to focus on the semantic content and thereby partially forego syntactic information. We argue that it is preferable to keep the original word order …