Bayesian Analysis of Dynamic Linear Topic Models
In dynamic topic modeling, the proportional contribution of a topic to a document depends on the temporal dynamics of that topic's overall prevalence in the corpus. We extend the Dynamic Topic Model of Blei and Lafferty (2006) by explicitly modeling document level topic proportions with covariates and dynamic structure that includes polynomial trends and periodicity. A Markov Chain Monte Carlo (MCMC) algorithm that utilizes Polya-Gamma data augmentation is developed for posterior inference. Conditional independencies in the model and sampling are made explicit, and our MCMC algorithm is parallelized where possible to allow for inference in large corpora. To address computational bottlenecks associated with Polya-Gamma sampling, we appeal to the Central Limit Theorem to develop a Gaussian approximation to the Polya-Gamma random variable. This approximation is fast and reliable for parameter values relevant in the text mining domain. Our model and inference algorithm are validated with multiple simulation examples, and we consider the application of modeling trends in PubMed abstracts. We demonstrate that sharing information across documents is critical for accurately estimating document-specific topic proportions. We also show that explicitly modeling polynomial and periodic behavior improves our ability to predict topic prevalence at future time points.
Code (0)
등록된 구현이 없습니다.
Tasks
Data AugmentationDynamic Topic ModelingTopic ModelsSimilar Papers 제목 키워드 기반
A Bayesian approach to modeling topic-metadata relationships
The objective of advanced topic modeling is not only to explore latent topical structures, but also to estimate relationships between the discovered topics and theoretically relevant metadata. Methods used to estimate su…
regressionTopic ModelsA General Method for Robust Bayesian Modeling
Robust Bayesian models are appealing alternatives to standard models, providing protection from data that contains outliers or other departures from the model assumptions. Historically, robust models were mostly develope…
regressionTopic ModelsLearning Topic Models and Latent Bayesian Networks Under Expansion Constraints
Unsupervised estimation of latent variable models is a fundamental problem central to numerous applications of machine learning and statistics. This work presents a principled approach for estimating broad classes of suc…
Topic ModelsExploring Topic-Metadata Relationships with the STM: A Bayesian Approach
The initial purpose of topic models was to identify latent topical clusters within unstructured text. Meanwhile, the focus of advanced studies has changed primarily to estimating the relationship between the discovered …
regressionTopic ModelsLearning document embeddings along with their uncertainties
Majority of the text modelling techniques yield only point-estimates of document embeddings and lack in capturing the uncertainty of the estimates. These uncertainties give a notion of how well the embeddings represent a…
Topic ModelsVariational Inference