More Than Words: Collocation Retokenization for Latent Dirichlet Allocation Models
Traditionally, Latent Dirichlet Allocation (LDA) ingests words in a collection of documents to discover their latent topics using word-document co-occurrences. Previous studies show that representing bigrams collocations in the input can improve topic coherence in English. However, it is unclear how to achieve the best results for languages without marked word boundaries such as Chinese and Thai. Here, we explore the use of retokenization based on chi-squared measures, t-statistics, and raw frequency to merge frequent token ngrams into collocations when preparing input to the LDA model. Based on the goodness of fit and the coherence metric, we show that topics trained with merged tokens result in topic keys that are clearer, more coherent, and more effective at distinguishing topics than those of unmerged models.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
More Than Words: Collocation Retokenization for Latent Dirichlet Allocation Models
Traditionally, Latent Dirichlet Allocation (LDA) ingests words in a collection of documents to discover their latent topics using word-document co-occurrences. Previous studies show that representing bigrams collocations…
Evaluation of Classification Algorithms and Features for Collocation Extraction in Croatian
Collocations can be defined as words that occur together significantly more often than it would be expected by chance. Many natural language processing applications such as natural language generation, word sense disambi…
General ClassificationKeyword ExtractionMachine TranslationPOS+3More Than Words: Collocation Tokenization for Latent Dirichlet Allocation Models
Traditionally, Latent Dirichlet Allocation (LDA) ingests words in a collection of documents to discover their latent topics using word-document co-occurrences. However, it is unclear how to achieve the best results for l…
Handling Collocations in Hierarchical Latent Tree Analysis for Topic Modeling
Topic modeling has been one of the most active research areas in machine learning in recent years. Hierarchical latent tree analysis (HLTA) has been recently proposed for hierarchical topic modeling and has shown superio…
Emergent retokenization symmetry in large language models: phenomenology and applications
Tokenization introduces representational redundancy: under a fixed token vocabulary, every byte string admits many valid token encodings, or segmentations, that decode to the same surface string. However, given a prompt,…