Inference in topic models: sparsity and trade-off
Topic models are popular for modeling discrete data (e.g., texts, images, videos, links), and provide an efficient way to discover hidden structures/semantics in massive data. One of the core problems in this field is the posterior inference for individual data instances. This problem is particularly important in streaming environments, but is often intractable. In this paper, we investigate the use of the Frank-Wolfe algorithm (FW) for recovering sparse solutions to posterior inference. From detailed elucidation of both theoretical and practical aspects, FW exhibits many interesting properties which are beneficial to topic modeling. We then employ FW to design fast methods, including ML-FW, for learning latent Dirichlet allocation (LDA) at large scales. Extensive experiments show that to reach the same predictiveness level, ML-FW can perform tens to thousand times faster than existing state-of-the-art methods for learning LDA from massive/streaming data.
Code (1)
Tasks
Topic ModelsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Managing sparsity, time, and quality of inference in topic models
Inference is an integral part of probabilistic topic models, but is often non-trivial to derive an efficient algorithm for a specific model. It is even much more challenging when we want to find a fast inference algorith…
Topic ModelsAutoencoding Documents for Topic Modeling with L-2 Sparsity Regularization
We propose a novel yet simple neural network architecture for topic modelling. The method is based on training an autoencoder structure where the bottleneck represents the space of the topics distribution and the decoder…
DecoderNeural Sparse Topical Coding
Topic models with sparsity enhancement have been proven to be effective at learning discriminative and coherent latent topics of short texts, which is critical to many scientific and engineering applications. However, th…
Language ModelingLanguage ModellingTopic ModelsWord EmbeddingsSparsemax and Relaxed Wasserstein for Topic Sparsity
Topic sparsity refers to the observation that individual documents usually focus on several salient topics instead of covering a wide variety of topics, and a real topic adopts a narrow range of terms instead of a wide c…
PLUM: Improving Inference Efficiency By Leveraging Repetition-Sparsity Trade-Off
Efficient inference of Deep Neural Networks (DNNs) on resource-constrained edge devices is essential. Quantization and sparsity are key techniques that translate to repetition and sparsity within tensors at the hardware-…
BinarizationComputational EfficiencyQuantizationRepresentation Learning