Revisiting Automated Topic Model Evaluation with Large Language Models
Topic models are used to make sense of large text collections. However, automatically evaluating topic model output and determining the optimal number of topics both have been longstanding challenges, with no effective automated solutions to date. This paper proposes using large language models to evaluate such output. We find that large language models appropriately assess the resulting topics, correlating more strongly with human judgments than existing automated metrics. We then investigate whether we can use large language models to automatically determine the optimal number of topics. We automatically assign labels to documents and choosing configurations with the most pure labels returns reasonable values for the optimal number of topics.
Code (1)
Tasks
Topic ModelsSimilar Papers 제목 키워드 기반
Toward Purpose-oriented Topic Model Evaluation enabled by Large Language Models
This study presents a framework for automated evaluation of dynamically evolving topic models using Large Language Models (LLMs). Topic modeling is essential for organizing and retrieving scholarly content in digital lib…
Topic ModelsPlug-and-Blend: A Framework for Controllable Story Generation with Blended Control Codes
Large pre-trained neural language models (LM) have very powerful text generation capabilities. However, in practice, they are hard to control for creative purposes. We describe a Plug-and-Play controllable language gener…
Story GenerationText GenerationContextualized Topic Coherence Metrics
The recent explosion in work on neural topic modeling has been criticized for optimizing automated topic evaluation metrics at the expense of actual meaningful topic identification. But human annotation remains expensive…
Topic ModelsRevisiting Topic-Guided Language Models
A recent line of work in natural language processing has aimed to combine language models and topic models. These topic-guided language models augment neural language models with topic models, unsupervised learning metho…
Language ModelingLanguage ModellingTopic ModelsIs Automated Topic Model Evaluation Broken? The Incoherence of Coherence
Topic model evaluation, like evaluation of other unsupervised methods, can be contentious. However, the field has coalesced around automated estimates of topic coherence, which rely on the frequency of word co-occurrence…
Topic Models