Seeing through the Human Reporting Bias: Visual Classifiers from Noisy Human-Centric Labels
When human annotators are given a choice about what to label in an image, they apply their own subjective judgments on what to ignore and what to mention. We refer to these noisy "human-centric" annotations as exhibiting human reporting bias. Examples of such annotations include image tags and keywords found on photo sharing sites, or in datasets containing image captions. In this paper, we use these noisy annotations for learning visually correct image classifiers. Such annotations do not use consistent vocabulary, and miss a significant amount of the information present in an image; however, we demonstrate that the noise in these annotations exhibits structure and can be modeled. We propose an algorithm to decouple the human reporting bias from the correct visually grounded labels. Our results are highly interpretable for reporting "what's in the image" versus "what's worth saying." We demonstrate the algorithm's efficacy along a variety of metrics and datasets, including MS COCO and Yahoo Flickr 100M. We show significant improvements over traditional algorithms for both image classification and image captioning, doubling the performance of existing methods in some cases.
Code (0)
등록된 구현이 없습니다.
Tasks
Image Captioningimage-classificationImage ClassificationSimilar Papers 제목 키워드 기반
When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs
Visual grounding is typically evaluated as a one-shot mapping from an informative referring expression to a visual target. This formulation misses a central property of real-world reference: initial referring expressions…
Referring ExpressionVisual GroundingVision: looking and seeing through our brain's information bottleneck
Our brain recognizes only a tiny fraction of sensory input, due to an information processing bottleneck. This blinds us to most visual inputs. Since we are blind to this blindness, only a recent framework highlights this…
Seeing the Goal, Missing the Truth: Human Accountability for AI Bias
This research explores how human-defined goals influence the behavior of Large Language Models (LLMs) through purpose-conditioned cognition. Using financial prediction tasks, we show that revealing the downstream use (e.…
Enhancing Zero-shot Commonsense Reasoning by Integrating Visual Knowledge via Machine Imagination
Recent advancements in zero-shot commonsense reasoning have empowered Pre-trained Language Models (PLMs) to acquire extensive commonsense knowledge without requiring task-specific fine-tuning. Despite this progress, thes…
Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute association
Reporting bias arises when people assume that some knowledge is universally understood and hence, do not necessitate explicit elaboration. In this paper, we focus on the wide existence of reporting bias in visual-languag…
AttributeObject