Testing High-dimensional Multinomials with Applications to Text Analysis
Motivated by applications in text mining and discrete distribution inference, we investigate the testing for equality of probability mass functions of $K$ groups of high-dimensional multinomial distributions. A test statistic, which is shown to have an asymptotic standard normal distribution under the null, is proposed. The optimal detection boundary is established, and the proposed test is shown to achieve this optimal detection boundary across the entire parameter space of interest. The proposed method is demonstrated in simulation studies and applied to analyze two real-world datasets to examine variation among consumer reviews of Amazon movies and diversity of statistical paper abstracts.
Code (0)
등록된 구현이 없습니다.
Tasks
DiversityVocal Bursts Intensity PredictionMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Hypothesis Testing for High-Dimensional Multinomials: A Selective Review
The statistical analysis of discrete data has been the subject of extensive statistical research dating back to the work of Pearson. In this survey we review some recently developed methods for testing hypotheses about h…
Two-sample testingVocal Bursts Intensity PredictionHypothesis Testing For Densities and High-Dimensional Multinomials: Sharp Local Minimax Rates
We consider the goodness-of-fit testing problem of distinguishing whether the data are drawn from a specified distribution, versus a composite alternative separated from the null in the total variation metric. In the dis…
Two-sample testingAdjusted chi-square test for degree-corrected block models
We propose a goodness-of-fit test for degree-corrected stochastic block models (DCSBM). The test is based on an adjusted chi-square statistic for measuring equality of means among groups of $n$ multinomial distributions …
Computational EfficiencyFast Inference for Interactive Models of Text
Probabilistic models are a useful means for analyzing large text corpora. Integrating such models with human interaction enables many new use cases. However, adding human interaction to probabilistic models requires infe…
ClusteringTopic ModelsMinimax optimal goodness-of-fit testing for densities and multinomials under a local differential privacy constraint
Finding anonymization mechanisms to protect personal data is at the heart of recent machine learning research. Here, we consider the consequences of local differential privacy constraints on goodness-of-fit testing, i.e.…