Estimating Jaccard Index with Missing Observations: A Matrix Calibration Approach
The Jaccard index is a standard statistics for comparing the pairwise similarity between data samples. This paper investigates the problem of estimating a Jaccard index matrix when there are missing observations in data samples. Starting from a Jaccard index matrix approximated from the incomplete data, our method calibrates the matrix to meet the requirement of positive semi-definiteness and other constraints, through a simple alternating projection algorithm. Compared with conventional approaches that estimate the similarity matrix based on the imputed data, our method has a strong advantage in that the calibrated matrix is guaranteed to be closer to the unknown ground truth in the Frobenius norm than the un-calibrated matrix (except in special cases they are identical). We carried out a series of empirical experiments and the results confirmed our theoretical justification. The evaluation also reported significantly improved results in real learning tasks on benchmarked datasets.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
One-sided Matrix Completion from Two Observations Per Row
Given only a few observed entries from a low-rank matrix $X$, matrix completion is the problem of imputing the missing entries, and it formalizes a wide range of real-world settings that involve estimating missing data. …
Matrix CompletionMissing ValuesEntrywise Inference for Missing Panel Data: A Simple and Instance-Optimal Approach
Longitudinal or panel data can be represented as a matrix with rows indexed by units and columns indexed by time. We consider inferential questions associated with the missing data version of panel data induced by stagge…
Causal InferenceDenoisingA Distance Measure for Random Permutation Set: From the Layer-2 Belief Structure Perspective
Random permutation set (RPS) is a recently proposed framework designed to represent order-structured uncertain information. Measuring the distance between permutation mass functions is a key research topic in RPS theory …
Decomposing the Jaccard Distance and the Jaccard Index in ABCDE
ABCDE is a sophisticated technique for evaluating differences between very large clusterings. Its main metric that characterizes the magnitude of the difference between two clusterings is the JaccardDistance, which is a …
ClusteringInference for Sparse Conditional Precision Matrices
Given $n$ i.i.d. observations of a random vector $(X,Z)$, where $X$ is a high-dimensional vector and $Z$ is a low-dimensional index variable, we study the problem of estimating the conditional inverse covariance matrix $…