Minimax Optimal Estimation of KL Divergence for Continuous Distributions
Estimating Kullback-Leibler divergence from identical and independently distributed samples is an important problem in various domains. One simple and effective estimator is based on the k nearest neighbor distances between these samples. In this paper, we analyze the convergence rates of the bias and variance of this estimator. Furthermore, we derive a lower bound of the minimax mean square error and show that kNN method is asymptotically rate optimal.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Exactly Minimax-Optimal Locally Differentially Private Sampling
The sampling problem under local differential privacy has recently been studied with potential applications to generative models, but a fundamental analysis of its privacy-utility trade-off (PUT) remains incomplete. In t…
Estimation of discrete distributions with high probability under $χ^2$-divergence
We investigate the high-probability estimation of discrete distributions from an \iid sample under $χ^2$-divergence loss. Although the minimax risk in expectation is well understood, its high-probability counterpart rema…
Instance-Optimality for Private KL Distribution Estimation
We study the fundamental problem of estimating an unknown discrete distribution $p$ over $d$ symbols, given $n$ i.i.d. samples from the distribution. We are interested in minimizing the KL divergence between the true dis…
Nonparametric Estimation of Renyi Divergence and Friends
We consider nonparametric estimation of $L_2$, Renyi-$\alpha$ and Tsallis-$\alpha$ divergences between continuous distributions. Our approach is to construct estimators for particular integral functionals of two densitie…
On Estimating $L_2^2$ Divergence
We give a comprehensive theoretical characterization of a nonparametric estimator for the $L_2^2$ divergence between two continuous distributions. We first bound the rate of convergence of our estimator, showing that it …