"Is a picture of a bird a bird": Policy recommendations for dealing with ambiguity in machine vision models
Many questions that we ask about the world do not have a single clear answer, yet typical human annotation set-ups in machine learning assume there must be a single ground truth label for all examples in every task. The divergence between reality and practice is stark, especially in cases with inherent ambiguity and where the range of different subjective judgments is wide. Here, we examine the implications of subjective human judgments in the behavioral task of labeling images used to train machine vision models. We identify three primary sources of ambiguity arising from (i) depictions of labels in the images, (ii) raters' backgrounds, and (iii) the task definition. On the basis of the empirical results, we suggest best practices for handling label ambiguity in machine learning datasets.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Towards Building a Real Time Mobile Device Bird Counting System Through Synthetic Data Training and Model Compression
Counting the number of birds in an open sky setting has been an challenging problem due to the large number of bird flocks and the birds can overlap. Another difficulty is the lack of accurate training samples since the …
Crowd CountingModel CompressionLearning Extreme Hummingbird Maneuvers on Flapping Wing Robots
Biological studies show that hummingbirds can perform extreme aerobatic maneuvers during fast escape. Given a sudden looming visual stimulus at hover, a hummingbird initiates a fast backward translation coupled with a 18…
reinforcement-learningReinforcement LearningReinforcement Learning (RL)LittleBird: Efficient Faster & Longer Transformer for Question Answering
BERT has shown a lot of sucess in a wide variety of NLP tasks. But it has a limitation dealing with long inputs due to its attention mechanism. Longformer, ETC and BigBird addressed this issue and effectively solved the …
Language ModellingQuestion AnsweringFlapAI Bird: Training an Agent to Play Flappy Bird Using Reinforcement Learning Techniques
Reinforcement learning is one of the most popular approaches for automated game playing. This method allows an agent to estimate the expected utility of its state in order to make optimal actions in an unknown environmen…
Q-Learningreinforcement-learningReinforcement LearningReinforcement Learning (RL)Bird Movement Prediction Using Long Short-Term Memory Networks to Prevent Bird Strikes with Low Altitude Aircraft
The number of collisions between aircraft and birds in the airspace has been increasing at an alarming rate over the past decade due to increasing bird population, air traffic and usage of quieter aircraft. Bird strikes …