ABCDE: Application-Based Cluster Diff Evals
This paper considers the problem of evaluating clusterings of very large populations of items. Given two clusterings, namely a Baseline clustering and an Experiment clustering, the tasks are twofold: 1) characterize their differences, and 2) determine which clustering is better. ABCDE is a novel evaluation technique for accomplishing that. It aims to be practical: it allows items to have associated importance values that are application-specific, it is frugal in its use of human judgements when determining which clustering is better, and it can report metrics for arbitrary slices of items, thereby facilitating understanding and debugging. The approach to measuring the delta in the clustering quality is novel: instead of trying to construct an expensive ground truth up front and evaluating the each clustering with respect to that, where the ground truth must effectively pre-anticipate clustering changes, ABCDE samples questions for judgement on the basis of the actual diffs between the clusterings. ABCDE builds upon the pointwise metrics for clustering evaluation, which make the ABCDE metrics intuitive and simple to understand. The mathematical elegance of the pointwise metrics equip ABCDE with rigorous yet practical ways to explore the clustering diffs and to estimate the quality delta.
Code (0)
등록된 구현이 없습니다.
Tasks
ClusteringSimilar Papers 제목 키워드 기반
More Clustering Quality Metrics for ABCDE
ABCDE is a technique for evaluating clusterings of very large populations of items. Given two clusterings, namely a Baseline clustering and an Experiment clustering, ABCDE can characterize their differences with impact a…
ClusteringEvaluation of Cluster Id Assignment Schemes with ABCDE
A cluster id assignment scheme labels each cluster of a clustering with a distinct id. The goal of id assignment is semantic id stability, which means that, whenever possible, a cluster for the same underlying concept as…
ClusteringDecomposing the Jaccard Distance and the Jaccard Index in ABCDE
ABCDE is a sophisticated technique for evaluating differences between very large clusterings. Its main metric that characterizes the magnitude of the difference between two clusterings is the JaccardDistance, which is a …
ClusteringAmortized Bayesian Causal Discovery of Extended Factor Graphs
Learning causal graphs from interventional data is a challenging problem with broad applications. In molecular biology, for example, a central goal is to uncover gene regulatory networks from large-scale perturbation dat…
Bayesian InferenceABCDE: An Agent-Based Cognitive Development Environment
Children's cognitive abilities are sometimes cited as AI benchmarks. How can the most common 1,000 concepts (89\% of everyday use) be learnt in a naturalistic children's setting? Cognitive development in children is abou…