A jury evaluation theorem
Majority voting (MV) is the prototypical ``wisdom of the crowd'' algorithm. Theorems considering when MV is optimal for group decisions date back to Condorcet's 1785 jury decision theorem. The same assumption of error independence used by Condorcet is used here to prove a jury evaluation theorem that does purely algebraic evaluation (AE). Three or more binary jurors are enough to obtain the only two possible statistics of their correctness on a joint test they took. AE is shown to be superior to MV since it allows one to choose the minority vote depending on how the jurors agree or disagree. In addition, AE is self-alarming about the failure of the error-independence assumption. Experiments labeling demographic datasets from the American Community Survey are carried out to compare MV and AE on nearly error-independent ensembles. In general, using algebraic evaluation leads to better classifier evaluations and group labeling decisions.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Condorcet's Jury Theorem for Consensus Clustering and its Implications for Diversity
Condorcet's Jury Theorem has been invoked for ensemble classifiers to indicate that the combination of many classifiers can have better predictive performance than a single classifier. Such a theoretical underpinning is …
ClusteringDiversityMajority Voting and the Condorcet's Jury Theorem
There is a striking relationship between a three hundred years old Political Science theorem named "Condorcet's jury theorem" (1785), which states that majorities are more likely to choose correctly when individual votes…
BIG-bench Machine LearningEnsemble LearningSocial Media and Democracy
We study the ability of a social media platform with a political agenda to influence voting outcomes. Our benchmark is Condorcet's jury theorem, which states that the likelihood of a correct decision under majority votin…
On the probability of the Condorcet Jury Theorem or the Miracle of Aggregation
The Condorcet Jury Theorem or the Miracle of Aggregation are frequently invoked to ensure the competence of some aggregate decision-making processes. In this article we explore an estimation of the prior probability of t…
Decision MakingWho Reviews The Reviewers? A Multi-Level Jury Problem
We consider the problem of determining a binary ground truth using advice from a group of independent reviewers (experts) who express their guess about a ground truth correctly with some independent probability (competen…