Gamma Mixture Modeling for Cosine Similarity in Small Language Models

2025-10-06 · Kevin Player arxiv

We study the cosine similarity of sentence transformer embeddings and observe that they are well modeled by gamma mixtures. From a fixed corpus, we measure similarities between all document embeddings and a reference query embedding. Empirically we find that these distributions are often well captured by a gamma distribution shifted and truncated to [-1,1], and in many cases, by a gamma mixture. We propose a heuristic model in which a hierarchical clustering of topics naturally leads to a gamma-mixture structure in the similarity scores. Finally, we outline an expectation-maximization algorithm for fitting shifted gamma mixtures, which provides a practical tool for modeling similarity distributions.

📄 PDF Abstract BibTeX arXiv:2510.05309

Code (0)

등록된 구현이 없습니다.

Integrating Topic Modeling with Word Embeddings by Mixtures of vMFs

2016-12-01 · COLING 2016 12 · Xi-Ming Li, Jinjin Chi, Changchun Li, Jihong Ouyang 외

Gaussian LDA integrates topic modeling with word embeddings by replacing discrete topic distribution over word types with multivariate Gaussian distribution on the embedding space. This can take semantic information of w…

Topic Models Word Embeddings

Variational Mixture Models with Gamma or inverse-Gamma components

2016-07-26 · A. Llera, D. Vidaurre, R. H. R. Pruim, C. F. Beckmann

Mixture models with Gamma and or inverse-Gamma distributed mixture components are useful for medical image tissue segmentation or as post-hoc models for regression coefficients obtained from linear regression within a Ge…

blind source separation regression

Cosine-Similarity Routing with Semantic Anchors for Interpretable Mixture-of-Experts Language Models

2025-09-12 · Ivan Ternovtsii, Yurii Bilak arxiv

Mixture-of-Experts (MoE) models improve efficiency through sparse activation, but their learned gating functions provide limited insight into routing decisions. This work introduces the Semantic Resonance Architecture (S…

Negative Binomial Process Count and Mixture Modeling

2012-09-15 · Mingyuan Zhou, Lawrence Carin

The seemingly disjoint problems of count and mixture modeling are united under the negative binomial (NB) process. A gamma process is employed to model the rate measure of a Poisson process, whose normalization provides …

Bayesian Inference

Inference and Mixture Modeling with the Elliptical Gamma Distribution

2014-10-17 · Reshad Hosseini, Suvrit Sra, Lucas Theis, Matthias Bethge

We study modeling and inference with the Elliptical Gamma Distribution (EGD). We consider maximum likelihood (ML) estimation for EGD scatter matrices, a task for which we develop new fixed-point algorithms. Our algorithm…

Density Estimation