Evaluating Crowdsourcing Participants in the Absence of Ground-Truth
Given a supervised/semi-supervised learning scenario where multiple annotators are available, we consider the problem of identification of adversarial or unreliable annotators.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Evaluating LLM-corrupted Crowdsourcing Data Without Ground Truth
The recent success of generative AI highlights the crucial role of high-quality human feedback in building trustworthy AI systems. However, the increasing use of large language models (LLMs) by crowdsourcing workers pose…
Multiple-choiceFairness Evaluation for Uplift Modeling in the Absence of Ground Truth
The acceleration in the adoption of AI-based automated decision-making systems poses a challenge for evaluating the fairness of algorithmic decisions, especially in the absence of ground truth. When designing interventio…
counterfactualDecision MakingFairnessMarketingWeak Supervision Performance Evaluation via Partial Identification
Programmatic Weak Supervision (PWS) enables supervised model training without direct access to ground truth labels, utilizing weak labels from heuristics, crowdsourcing, or pre-trained models. However, the absence of gro…
Weakly-supervised Learning``Fingers in the Nose'': Evaluating Speakers' Identification of Multi-Word Expressions Using a Slightly Gamified Crowdsourcing Platform
This article presents the results we obtained in crowdsourcing French speakers{'} intuition concerning multi-work expressions (MWEs). We developed a slightly gamified crowdsourcing platform, part of which is designed to …
Inferring the ground truth through crowdsourcing
Universally valid ground truth is almost impossible to obtain or would come at a very high cost. For supervised learning without universally valid ground truth, a recommended approach is applying crowdsourcing: Gathering…
Mitosis Detectionvalid