CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation
Human annotated data plays a crucial role in machine learning (ML) research and development. However, the ethical considerations around the processes and decisions that go into dataset annotation have not received nearly enough attention. In this paper, we survey an array of literature that provides insights into ethical considerations around crowdsourced dataset annotation. We synthesize these insights, and lay out the challenges in this space along two layers: (1) who the annotator is, and how the annotators' lived experiences can impact their annotations, and (2) the relationship between the annotators and the crowdsourcing platforms, and what that relationship affords them. Finally, we introduce a novel framework, CrowdWorkSheets, for dataset developers to facilitate transparent documentation of key decisions points at various stages of the data annotation pipeline: task formulation, selection of annotators, platform and infrastructure choices, dataset analysis and evaluation, and dataset release and maintenance.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Intrinsic statistical separation of subpopulations in heterogeneous collective motion via dimensionality reduction
Collective motion of locally interacting agents is found ubiquitously throughout nature. The inability to probe individuals has driven longstanding interest in the development of methods for inferring the underlying inte…
ClusteringDimensionality ReductionDisentanglementWhose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotation
Human annotations play a crucial role in machine learning (ML) research and development. However, the ethical considerations around the processes and decisions that go into building ML datasets has not received nearly en…
idtracker.ai: Tracking all individuals in large collectives of unmarked animals
Our understanding of collective animal behavior is limited by our ability to track each of the individuals. We describe an algorithm and software, idtracker.ai, that extracts from video all trajectories with correct iden…
AllMulti-Animal Tracking with identificationLoose social organisation of AB strain zebrafish groups in a two-patch environment
We explore the collective behaviours of 7 group sizes: 1, 2, 3, 5, 7, 10 and 20 AB zebrafish (Danio rerio) in a constraint environment composed of two identical squared rooms connected by a corridor. This simple set-up i…
AIR: Unifying Individual and Collective Exploration in Cooperative Multi-Agent Reinforcement Learning
Exploration in cooperative multi-agent reinforcement learning (MARL) remains challenging for value-based agents due to the absence of an explicit policy. Existing approaches include individual exploration based on uncert…
DiversityMulti-agent Reinforcement Learning