Dependent Multinomial Models Made Easy: Stick Breaking with the Pólya-Gamma Augmentation
Many practical modeling problems involve discrete data that are best represented as draws from multinomial or categorical distributions. For example, nucleotides in a DNA sequence, children's names in a given state and year, and text documents are all commonly modeled with multinomial distributions. In all of these cases, we expect some form of dependency between the draws: the nucleotide at one position in the DNA strand may depend on the preceding nucleotides, children's names are highly correlated from year to year, and topics in text may be correlated and dynamic. These dependencies are not naturally captured by the typical Dirichlet-multinomial formulation. Here, we leverage a logistic stick-breaking representation and recent innovations in P\'olya-gamma augmentation to reformulate the multinomial distribution in terms of latent variables with jointly Gaussian likelihoods, enabling us to take advantage of a host of Bayesian inference techniques for Gaussian models with minimal overhead.
Code (1)
Tasks
Bayesian InferencePositionSimilar Papers 제목 키워드 기반
Dependent Multinomial Models Made Easy: Stick-Breaking with the Polya-gamma Augmentation
Many practical modeling problems involve discrete data that are best represented as draws from multinomial or categorical distributions. For example, nucleotides in a DNA sequence, children's names in a given state and y…
Bayesian InferencePermuted and Augmented Stick-Breaking Bayesian Multinomial Regression
To model categorical response variables given their covariates, we propose a permuted and augmented stick-breaking (paSB) construction that one-to-one maps the observed categories to randomly permuted latent sticks. This…
regressionA Conditional Multinomial Mixture Model for Superset Label Learning
In the superset label learning problem (SLL), each training instance provides a set of candidate labels of which one is the true label of the instance. As in ordinary regression, the candidate label set is a noisy versio…
An elementary derivation of the Chinese restaurant process from Sethuraman's stick-breaking process
The Chinese restaurant process (CRP) and the stick-breaking process are the two most commonly used representations of the Dirichlet process. However, the usual proof of the connection between them is indirect, relying on…
Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Study
The self-attention mechanism traditionally relies on the softmax operator, necessitating positional embeddings like RoPE, or position biases to account for token order. But current methods using still face length general…