MiSC: Mixed Strategies Crowdsourcing
Popular crowdsourcing techniques mostly focus on evaluating workers' labeling quality before adjusting their weights during label aggregation. Recently, another cohort of models regard crowdsourced annotations as incomplete tensors and recover unfilled labels by tensor completion. However, mixed strategies of the two methodologies have never been comprehensively investigated, leaving them as rather independent approaches. In this work, we propose $\textit{MiSC}$ ($\textbf{Mi}$xed $\textbf{S}$trategies $\textbf{C}$rowdsourcing), a versatile framework integrating arbitrary conventional crowdsourcing and tensor completion techniques. In particular, we propose a novel iterative Tucker label aggregation algorithm that outperforms state-of-the-art methods in extensive experiments.
Code (0)
등록된 구현이 없습니다.
Methods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Crowdsourcing the Perception of Machine Teaching
Teachable interfaces can empower end-users to attune machine learning systems to their idiosyncratic characteristics and environment by explicitly providing pertinent training examples. While facilitating control, their …
DiversityMisconceptionsExact Exponent in Optimal Rates for Crowdsourcing
In many machine learning applications, crowdsourcing has become the primary means for label collection. In this paper, we study the optimal error rate for aggregating labels provided by a set of non-expert workers. Under…
Heterogeneous Noise and Stable Miscoordination
Coordination games admit two types of equilibria: pure equilibria, where all players successfully coordinate their actions, and mixed equilibria, where players frequently experience miscoordination. The existing literatu…
Word Sense Inventories by Non-Experts.
In this paper, we explore different strategies for implementing a crowdsourcing methodology for a single-step construction of an empirically-derived sense inventory and the corresponding sense-annotated corpus. We report…
Word Sense DisambiguationYour 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings
Cardinal scores (numeric ratings) collected from people are well known to suffer from miscalibrations. A popular approach to address this issue is to assume simplistic models of miscalibration (such as linear biases) to …