On the Kullback-Leibler divergence between pairwise isotropic Gaussian-Markov random fields
The Kullback-Leibler divergence or relative entropy is an information-theoretic measure between statistical models that play an important role in measuring a distance between random variables. In the study of complex systems, random fields are mathematical structures that models the interaction between these variables by means of an inverse temperature parameter, responsible for controlling the spatial dependence structure along the field. In this paper, we derive closed-form expressions for the Kullback-Leibler divergence between two pairwise isotropic Gaussian-Markov random fields in both univariate and multivariate cases. The proposed equation allows the development of novel similarity measures in image processing and machine learning applications, such as image denoising and unsupervised metric learning.
Code (0)
등록된 구현이 없습니다.
Tasks
DenoisingImage DenoisingMetric LearningSimilar Papers 제목 키워드 기반
A solvable generative model with a linear, one-step denoiser
We develop an analytically tractable single-step diffusion model based on a linear denoiser and present explicit formula for the Kullback-Leibler divergence between generated and sampling distribution, taken to be isotro…
MemorizationExclusive Topic Modeling
We propose an Exclusive Topic Modeling (ETM) for unsupervised text classification, which is able to 1) identify the field-specific keywords though less frequently appeared and 2) deliver well-structured topics with exclu…
text-classificationText ClassificationUnsupervised Text ClassificationGuaranteed bounds on the Kullback-Leibler divergence of univariate mixtures using piecewise log-sum-exp inequalities
Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks. Since the Kullback-Leibler divergence …
FormRényi Divergence and Kullback-Leibler Divergence
R\'enyi divergence is related to R\'enyi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by R\'enyi as a measure of information that satisfi…
Statistical and Geometrical properties of regularized Kernel Kullback-Leibler divergence
In this paper, we study the statistical and geometrical properties of the Kullback-Leibler divergence with kernel covariance operators (KKL) introduced by Bach [2022]. Unlike the classical Kullback-Leibler (KL) divergenc…