Efficient Competitions and Online Learning with Strategic Forecasters
Winner-take-all competitions in forecasting and machine-learning suffer from distorted incentives. Witkowski et al. 2018 identified this problem and proposed ELF, a truthful mechanism to select a winner. We show that, from a pool of $n$ forecasters, ELF requires $\Theta(n\log n)$ events or test data points to select a near-optimal forecaster with high probability. We then show that standard online learning algorithms select an $\epsilon$-optimal forecaster using only $O(\log(n) / \epsilon^2)$ events, by way of a strong approximate-truthfulness guarantee. This bound matches the best possible even in the nonstrategic setting. We then apply these mechanisms to obtain the first no-regret guarantee for non-myopic strategic experts.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
No-Regret Online Prediction with Strategic Experts
We study a generalization of the online binary prediction with expert advice framework where at each round, the learner is allowed to pick $m\geq 1$ experts from a pool of $K$ experts and the overall utility is a modular…
PredictionEvaluating Strategic Reasoning in Forecasting Agents
Forecasting benchmarks produce accuracy leaderboards but little insight into why some forecasters are more accurate than others. We introduce Bench to the Future 2 (BTF-2), 1,417 pastcasting questions with a frozen 15M-d…
Hierarchical Partitioning Forecaster
In this work we consider a new family of algorithms for sequential prediction, Hierarchical Partitioning Forecasters (HPFs). Our goal is to provide appealing theoretical - regret guarantees on a powerful model class - an…
The impacts of various parameters on learning process and machine learning based performance prediction in online coding competitions
Various parameters affect the performance of students in online coding competitions. Students' behavior, approach, emotions, and problem difficulty levels significantly impact their performance in online coding competiti…
BIG-bench Machine LearningComparing Sequential Forecasters
Consider two forecasters, each making a single prediction for a sequence of events over time. We ask a relatively basic question: how might we compare these forecasters, either online or post-hoc, while avoiding unverifi…
valid