Graph Contrastive Topic Model
Existing NTMs with contrastive learning suffer from the sample bias problem owing to the word frequency-based sampling strategy, which may result in false negative samples with similar semantics to the prototypes. In this paper, we aim to explore the efficient sampling strategy and contrastive learning in NTMs to address the aforementioned issue. We propose a new sampling assumption that negative samples should contain words that are semantically irrelevant to the prototype. Based on it, we propose the graph contrastive topic model (GCTM), which conducts graph contrastive learning (GCL) using informative positive and negative samples that are generated by the graph-based sampling strategy leveraging in-depth correlation and irrelevance among documents and words. In GCTM, we first model the input document as the document word bipartite graph (DWBG), and construct positive and negative word co-occurrence graphs (WCGs), encoded by graph neural networks, to express in-depth semantic correlation and irrelevance among words. Based on the DWBG and WCGs, we design the document-word information propagation (DWIP) process to perform the edge perturbation of DWBG, based on multi-hop correlations/irrelevance among documents and words. This yields the desired negative and positive samples, which will be utilized for GCL together with the prototypes to improve learning document topic representations and latent topics. We further show that GCL can be interpreted as the structured variational graph auto-encoder which maximizes the mutual information of latent topic representations of different perspectives on DWBG. Experiments on several benchmark datasets demonstrate the effectiveness of our method for topic coherence and document representation learning compared with existing SOTA methods.
Code (1)
Tasks
Contrastive LearningmodelRepresentation LearningMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization
Recently, neural topic models (NTMs) have been incorporated into pre-trained language models (PLMs), to capture the global semantic information for text summarization. However, in these methods, there remain limitations …
Contrastive LearningExtractive SummarizationLanguage ModelingLanguage Modelling+2CoSD: Collaborative Stance Detection with Contrastive Heterogeneous Topic Graph Learning
Stance detection seeks to identify the viewpoints of individuals either in favor or against a given target or a controversial topic. Current advanced neural models for stance detection typically employ fully parametric s…
Graph LearningStance DetectionTopic Modeling as Multi-Objective Contrastive Optimization
Recent representation learning approaches enhance neural topic models by optimizing the weighted linear combination of the evidence lower bound (ELBO) of the log-likelihood and the contrastive learning objective that con…
Contrastive LearningDiversityRepresentation LearningTopic ModelsCGCL: Collaborative Graph Contrastive Learning without Handcrafted Graph Data Augmentations
Unsupervised graph representation learning is a non-trivial topic. The success of contrastive methods in the unsupervised representation learning on structured data inspires similar attempts on the graph. Existing graph …
Contrastive LearningData AugmentationGraph Representation LearningRepresentation Learning+1GNN-Transformer Cooperative Architecture for Trustworthy Graph Contrastive Learning
Graph contrastive learning (GCL) has become a hot topic in the field of graph representation learning. In contrast to traditional supervised learning relying on a large number of labels, GCL exploits augmentation strateg…
Contrastive LearningGraph Representation LearningRepresentation Learning