Extrapolating the profile of a finite population
We study a prototypical problem in empirical Bayes. Namely, consider a population consisting of $k$ individuals each belonging to one of $k$ types (some types can be empty). Without any structural restrictions, it is impossible to learn the composition of the full population having observed only a small (random) subsample of size $m = o(k)$. Nevertheless, we show that in the sublinear regime of $m =\omega(k/\log k)$, it is possible to consistently estimate in total variation the \emph{profile} of the population, defined as the empirical distribution of the sizes of each type, which determines many symmetric properties of the population. We also prove that in the linear regime of $m=c k$ for any constant $c$ the optimal rate is $\Theta(1/\log k)$. Our estimator is based on Wolfowitz's minimum distance method, which entails solving a linear program (LP) of size $k$. We show that there is a single infinite-dimensional LP whose value simultaneously characterizes the risk of the minimum distance estimator and certifies its minimax optimality. The sharp convergence rate is obtained by evaluating this LP using complex-analytic techniques.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
From Human-Level AI Tales to AI Leveling Human Scales
Comparing AI models to "human level" is often misleading when benchmark scores are incommensurate or human baselines are drawn from a narrow population. To address this, we propose a framework that calibrates items again…
Transfer Estimates for Causal Effects across Heterogeneous Sites
We consider the problem of extrapolating treatment effects across heterogeneous populations (``sites"/``contexts"). We consider an idealized scenario in which the researcher observes cross-sectional data for a large numb…
Extrapolating false alarm rates in automatic speaker verification
Automatic speaker verification (ASV) vendors and corpus providers would both benefit from tools to reliably extrapolate performance metrics for large speaker populations without collecting new speakers. We address false …
Speaker VerificationThe maximum entropy mortality model: forecasting mortality using statistical moments Codes
The age-at-deathdistribution is arepresentation ofthe mortality experience in a population. Although it proves to be highly informative, it is often neglected when it comes to the practice of past or future mortality ass…
Time SeriesTime Series AnalysisLoss-function learning for digital tissue deconvolution
The gene expression profile of a tissue averages the expression profiles of all cells in this tissue. Digital tissue deconvolution (DTD) addresses the following inverse problem: Given the expression profile $y$ of a tiss…