Bayesian Online Learning for Consensus Prediction
Given a pre-trained classifier and multiple human experts, we investigate the task of online classification where model predictions are provided for free but querying humans incurs a cost. In this practical but under-explored setting, oracle ground truth is not available. Instead, the prediction target is defined as the consensus vote of all experts. Given that querying full consensus can be costly, we propose a general framework for online Bayesian consensus estimation, leveraging properties of the multivariate hypergeometric distribution. Based on this framework, we propose a family of methods that dynamically estimate expert consensus from partial feedback by producing a posterior over expert and model beliefs. Analyzing this posterior induces an interpretable trade-off between querying cost and classification performance. We demonstrate the efficacy of our framework against a variety of baselines on CIFAR-10H and ImageNet-16H, two large-scale crowdsourced datasets.
Code (0)
등록된 구현이 없습니다.
Tasks
PredictionSimilar Papers 제목 키워드 기반
Bayesian Consensus: Consensus Estimates from Miscalibrated Instruments under Heteroscedastic Noise
We consider the problem of aggregating predictions or measurements from a set of human forecasters, models, sensors or other instruments which may be subject to bias or miscalibration and random heteroscedastic noise. We…
Bayesian InferenceDistributed Bayesian Online Learning for Cooperative Manipulation
For tasks where the dynamics of multiple agents are physically coupled, e.g., in cooperative manipulation, the coordination between the individual agents becomes crucial, which requires exact knowledge of the interaction…
A Consensus-Bayesian Framework for Detecting Malicious Activity in Enterprise Directory Access Graphs
This work presents a consensus-based Bayesian framework to detect malicious user behavior in enterprise directory access graphs. By modeling directories as topics and users as agents within a multi-level interaction grap…
Consensus controller with safety guarantee: an application to the kinematic bicycle model
This paper proposes a consensus controller for multi-agent systems that can guarantee the agents' safety. The controller, built with the idea of output prediction and the Newton-Raphson method, achieves consensus for a c…
Autonomous VehiclesA Convergence Predictor Model for Consensus-based Decentralised Energy Markets
This letter introduces a convergence prediction model (CPM) for decentralized market clearing mechanisms. The CPM serves as a tool to detect potential cyber-attacks that affect the convergence of the consensus mechanism …