Understanding The Robustness of Self-supervised Learning Through Topic Modeling
Self-supervised learning has significantly improved the performance of many NLP tasks. However, how can self-supervised learning discover useful representations, and why is it better than traditional approaches such as probabilistic models are still largely unknown. In this paper, we focus on the context of topic modeling and highlight a key advantage of self-supervised learning - when applied to data generated by topic models, self-supervised learning can be oblivious to the specific model, and hence is less susceptible to model misspecification. In particular, we prove that commonly used self-supervised objectives based on reconstruction or contrastive samples can both recover useful posterior information for general topic models. Empirically, we show that the same objectives can perform on par with posterior inference using the correct model, while outperforming posterior inference using misspecified models.
Code (0)
등록된 구현이 없습니다.
Tasks
Self-Supervised LearningTopic ModelsSimilar Papers 제목 키워드 기반
Semi-Supervised Learning Approach to Discover Enterprise User Insights from Feedback and Support
With the evolution of the cloud and customer centric culture, we inherently accumulate huge repositories of textual reviews, feedback, and support data.This has driven enterprises to seek and research engagement patterns…
ClusteringCultural Vocal Bursts Intensity PredictionTopic ModelsTransfer LearningN-Critics: Self-Refinement of Large Language Models with Ensemble of Critics
We propose a self-correction mechanism for Large Language Models (LLMs) to mitigate issues such as toxicity and fact hallucination. This method involves refining model outputs through an ensemble of critics and the model…
FairnessHallucinationOne Objective for All Models --- Self-supervised Learning for Topic Models
Self-supervised learning has significantly improved the performance of many NLP tasks. In this paper, we highlight a key advantage of self-supervised learning - when applied to data generated by topic models, self-superv…
AllSelf-Supervised LearningTopic ModelsCan Self-Supervised Representation Learning Methods Withstand Distribution Shifts and Corruptions?
Self-supervised learning in computer vision aims to leverage the inherent structure and relationships within data to learn meaningful representations without explicit human annotation, enabling a holistic understanding o…
Contrastive LearningKnowledge DistillationRepresentation LearningSelf-Supervised LearningInteractive Topic Models with Optimal Transport
Topic models are widely used to analyze document collections. While they are valuable for discovering latent topics in a corpus when analysts are unfamiliar with the corpus, analysts also commonly start with an understan…
Topic Models