A Bayesian Evaluation Framework for Subjectively Annotated Visual Recognition Tasks
An interesting development in automatic visual recognition has been the emergence of tasks where it is not possible to assign objective labels to images, yet still feasible to collect annotations that reflect human judgements about them. Machine learning-based predictors for these tasks rely on supervised training that models the behavior of the annotators, i.e., what would the average person's judgement be for an image? A key open question for this type of work, especially for applications where inconsistency with human behavior can lead to ethical lapses, is how to evaluate the epistemic uncertainty of trained predictors, i.e., the uncertainty that comes from the predictor's model. We propose a Bayesian framework for evaluating black box predictors in this regime, agnostic to the predictor's internal structure. The framework specifies how to estimate the epistemic uncertainty that comes from the predictor with respect to human labels by approximating a conditional distribution and producing a credible interval for the predictions and their measures of performance. The framework is successfully applied to four image classification tasks that use subjective human judgements: facial beauty assessment, social attribute assignment, apparent age estimation, and ambiguous scene labeling.
Code (1)
Tasks
Age EstimationAttributeimage-classificationImage ClassificationOpen-Ended Question AnsweringScene LabelingSimilar Papers 제목 키워드 기반
Versatile Speech Databases for High Quality Synthesis for Basque
This paper presents three new speech databases for standard Basque. They are designed primarily for corpus-based synthesis but each database has its specific purpose: 1) AhoSyn: high quality speech synthesis (recorded al…
Emotional Speech SynthesisSpeech SynthesisVocal Bursts Intensity PredictionVoice ConversionSubjective performance evaluation of bitrate allocation strategies for MPEG and JPEG Pleno point cloud compression
The recent rise in interest in point clouds as an imaging modality has motivated standardization groups such as MPEG and JPEG Pleno to launch activities aiming at developing compression standards for point clouds. Lossy …
Can LLMs Generate Human-Like Wayfinding Instructions? Towards Platform-Agnostic Embodied Instruction Synthesis
We present a novel approach to automatically synthesize "wayfinding instructions" for an embodied robot agent. In contrast to prior approaches that are heavily reliant on human-annotated datasets designed exclusively for…
In-Context LearningQuestion AnsweringVisual Question AnsweringBayesian Optimization with Conformal Prediction Sets
Bayesian optimization is a coherent, ubiquitous approach to decision-making under uncertainty, with applications including multi-arm bandits, active learning, and black-box optimization. Bayesian optimization selects dec…
Active LearningBayesian OptimizationConformal PredictionDecision Making+4Rho-Perfect: Correlation Ceiling For Subjective Evaluation Datasets
Subjective ratings contain inherent noise that limits the model-human correlation, but this reliability issue is rarely quantified. In this paper, we present $ρ$-Perfect, a practical estimation of the highest achievable …