Crowdsourcing with Unsure Option
One of the fundamental problems in crowdsourcing is the trade-off between the number of the workers needed for high-accuracy aggregation and the budget to pay. For saving budget, it is important to ensure high quality of the crowd-sourced labels, hence the total cost on label collection will be reduced. Since the self-confidence of the workers often has a close relationship with their abilities, a possible way for quality control is to request the workers to return the labels only when they feel confident, by means of providing unsure option to them. On the other hand, allowing workers to choose unsure option also leads to the potential danger of budget waste. In this work, we propose the analysis towards understanding when providing the unsure option indeed leads to significant cost reduction, as well as how the confidence threshold is set. We also propose an online mechanism, which is alternative for threshold selection when the estimation of the crowd ability distribution is difficult.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Partial-Label Learning with a Reject Option
In real-world applications, one often encounters ambiguously labeled data, where different annotators assign conflicting class labels. Partial-label learning allows training classifiers in this weakly supervised setting,…
Autonomous DrivingPartial Label LearningCan Crowdsourcing Survive the LLM Era? A Community Survey on Human Data Collection
The widespread use of Large Language Models (LLMs) as writing tools challenges the validity of crowdsourced data, as crowdworkers may outsource tasks to models. To better understand how this is addressed, we surveyed 155…
Aggregate Modeling and Equilibrium Analysis of the Crowdsourcing Market for Autonomous Vehicles
Autonomous vehicles (AVs) have the potential of reshaping the human mobility in a wide variety of aspects. This paper focuses on a new possibility that the AV owners have the option of "renting" their AVs to a company, w…
Autonomous VehiclesLearning With Unsure Data for Medical Image Diagnosis
In image-based disease prediction, it can be hard to give certain cases a deterministic "disease/normal" label due to lack of enough information, e.g., at its early stage. We call such cases "unsure" data. Labeling such …
Disease PredictionSemEval-2015 Task 3: Answer Selection in Community Question Answering
Community Question Answering (cQA) provides new interesting research directions to the traditional Question Answering (QA) field, e.g., the exploitation of the interaction between users and the structure of related posts…
Answer SelectionCommunity Question AnsweringQuestion Answering