Telescoping Density-Ratio Estimation
Density-ratio estimation via classification is a cornerstone of unsupervised learning. It has provided the foundation for state-of-the-art methods in representation learning and generative modelling, with the number of use-cases continuing to proliferate. However, it suffers from a critical limitation: it fails to accurately estimate ratios p/q for which the two densities differ significantly. Empirically, we find this occurs whenever the KL divergence between p and q exceeds tens of nats. To resolve this limitation, we introduce a new framework, telescoping density-ratio estimation (TRE), that enables the estimation of ratios between highly dissimilar densities in high-dimensional spaces. Our experiments demonstrate that TRE can yield substantial improvements over existing single-ratio methods for mutual information estimation, representation learning and energy-based modelling.
Code (1)
Tasks
Density Ratio EstimationMutual Information EstimationRepresentation LearningSimilar Papers 제목 키워드 기반
ScoreMatchingRiesz: Score Matching for Debiased Machine Learning and Policy Path Estimation
We propose ScoreMatchingRiesz, a family of Riesz representer estimators based on score matching. The Riesz representer is a key nuisance component in debiased machine learning, enabling $\sqrt{n}$-consistent and asymptot…
On Imbalanced Regression with Hoeffding Trees
Many real-world applications generate continuous data streams for regression. Hoeffding trees and their variants have a long-standing tradition due to their effectiveness, either alone or as base models in broader ensemb…
Density EstimationBreaking the Lens of the Telescope: Online Relevance Estimation over Large Retrieval Sets
Advanced relevance models, such as those that use large language models (LLMs), provide highly accurate relevance estimations. However, their computational costs make them infeasible for processing large document corpora…
Re-RankingRetrievalIG-Lens: Exact Additive Probability Attribution Across Transformer Layers via Telescoping Integrated Gradients
We ask a simple question about decoder-only transformers: between which two layers is the probability of a predicted token actually produced? Existing layer-wise readout tools answer only approximately. The logit lens an…
Multi-fidelity Monte Carlo: a pseudo-marginal approach
Markov chain Monte Carlo (MCMC) is an established approach for uncertainty quantification and propagation in scientific applications. A key challenge in applying MCMC to scientific domains is computation: the target dens…
Uncertainty Quantification