Improving Neural Topic Models using Knowledge Distillation
Topic models are often used to identify human-interpretable topics to help make sense of large document collections. We use knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers. Our modular method can be straightforwardly applied with any neural topic model to improve topic quality, which we demonstrate using two models having disparate architectures, obtaining state-of-the-art topic coherence. We show that our adaptable framework not only improves performance in the aggregate over all estimated topics, as is commonly reported, but also in head-to-head comparisons of aligned topics.
Code (1)
Tasks
Knowledge DistillationTopic ModelsMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Improving Neural Topic Models with Wasserstein Knowledge Distillation
Topic modeling is a dominant method for exploring document collections on the web and in digital libraries. Recent approaches to topic modeling use pretrained contextualized language models and variational autoencoders. …
Knowledge DistillationTopic ModelsTopic Modeling for Maternal Health Using Reddit
This paper applies topic modeling to understand maternal health topics, concerns, and questions expressed in online communities on social networking sites. We examine Latent Dirichlet Analysis (LDA) and two state-of-the-…
Knowledge DistillationKDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation
In text classification tasks, fine tuning pretrained language models like BERT and GPT-3 yields competitive accuracy; however, both methods require pretraining on large text datasets. In contrast, general topic modeling …
ClassificationKnowledge Distillationtext-classificationText ClassificationHomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers
Knowledge distillation has been shown to be a powerful model compression approach to facilitate the deployment of pre-trained language models in practice. This paper focuses on task-agnostic distillation. It produces a c…
Knowledge DistillationModel CompressionTransfer LearningA Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models
This paper aims to provide a selective survey about knowledge distillation(KD) framework for researchers and practitioners to take advantage of it for developing new optimized models in the deep neural network field. To …
Knowledge DistillationModel CompressionSelf-Supervised LearningSurvey