Improving Contextualized Topic Models with Negative Sampling
Topic modeling has emerged as a dominant method for exploring large document collections. Recent approaches to topic modeling use large contextualized language models and variational autoencoders. In this paper, we propose a negative sampling mechanism for a contextualized topic model to improve the quality of the generated topics. In particular, during model training, we perturb the generated document-topic vector and use a triplet loss to encourage the document reconstructed from the correct document-topic vector to be similar to the input document and dissimilar to the document reconstructed from the perturbed vector. Experiments for different topic counts on three publicly available benchmark datasets show that in most cases, our approach leads to an increase in topic coherence over that of the baselines. Our model also achieves very high topic diversity.
Code (2)
Tasks
DiversityTopic ModelsTripletMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Evaluating Negative Sampling Approaches for Neural Topic Models
Negative sampling has emerged as an effective technique that enables deep learning models to learn better representations by introducing the paradigm of learn-to-compare. The goal of this approach is to add robustness to…
Document ClassificationTopic ModelsRefining Dimensions for Improving Clustering-based Cross-lingual Topic Models
Recent works in clustering-based topic models perform well in monolingual topic identification by introducing a pipeline to cluster the contextualized representations. However, the pipeline is suboptimal in identifying t…
ClusteringTopic ModelsCWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling
Most existing topic models rely on bag-of-words (BOW) representation, which limits their ability to capture word order information and leads to challenges with out-of-vocabulary (OOV) words in new documents. Contextualiz…
Document ClassificationLanguage ModellingNatural Language UnderstandingNER+3Short Text Topic Modeling with Topic Distribution Quantization and Negative Sampling Decoder
Topic models have been prevailing for many years on discovering latent semantics while modeling long documents. However, for short texts they generally suffer from data sparsity because of extremely limited word co-occur…
DecoderQuantizationTopic ModelsTopic Modeling with Contextualized Word Representation Clusters
Clustering token-level contextualized word representations produces output that shares many similarities with topic models for English text collections. Unlike clusterings of vocabulary-level word embeddings, the resulti…
ClusteringTopic ModelsWord Embeddings