Strictly Proper Scoring Mechanisms Without Expected Arbitrage
When eliciting forecasts from a group of experts, it is important to reward predictions so that market participants are incentivized to tell the truth. Existing mechanisms partially accomplish this but remain susceptible to groups of experts colluding to increase their expected reward, meaning that no aggregation of predictions can be fully trusted to represent the true beliefs of forecasters. This paper presents two novel scoring mechanisms which elicit truthful forecasts from any group of experts, even if they can collude or access each other's predictions. The key insight of this approach is a randomization component which maintains strict properness but prevents experts from coordinating dishonest reports in advance. These mechanisms are strictly proper and do not admit expected arbitrage, resolving an open question in the field.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Language Generation with Strictly Proper Scoring Rules
Language generation based on maximum likelihood estimation (MLE) has become the fundamental approach for text generation. Maximum likelihood estimation is typically performed by minimizing the log-likelihood loss, also k…
Language ModelingLanguage Modellingscoring ruleText GenerationInducing Honest Reporting Without Observing Outcomes: An Application to the Peer-Review Process
When eliciting opinions from a group of experts, traditional devices used to promote honest reporting assume that there is an observable future outcome. In practice, however, this assumption is not always reasonable. In …
Pandora's Regret: A Proper Scoring Rule for Evaluating Sequential Search
In sequential search, alternatives are tested until the true class is found. Standard proper scoring rules like log loss are local, ignoring the ranking of competitors and misaligning model evaluation with search utility…
When do prophets profit in prediction markets?
Prediction markets aggregate dispersed beliefs into prices that act as probabilistic forecasts of uncertain events. Classical theory establishes a clean equivalence between forecasting accuracy and trading profit, but on…
Diversity Enhanced Active Learning with Strictly Proper Scoring Rules
We study acquisition functions for active learning (AL) for text classification. The Expected Loss Reduction (ELR) method focuses on a Bayesian estimate of the reduction in classification error, recently updated with Mea…
Active LearningDiversitytext-classificationText Classification