Inducing Honest Reporting Without Observing Outcomes: An Application to the Peer-Review Process
When eliciting opinions from a group of experts, traditional devices used to promote honest reporting assume that there is an observable future outcome. In practice, however, this assumption is not always reasonable. In this paper, we propose a scoring method built on strictly proper scoring rules to induce honest reporting without assuming observable outcomes. Our method provides scores based on pairwise comparisons between the reports made by each pair of experts in the group. For ease of exposition, we introduce our scoring method by illustrating its application to the peer-review process. In order to do so, we start by modeling the peer-review process using a Bayesian model where the uncertainty regarding the quality of the manuscript is taken into account. Thereafter, we introduce our scoring method to evaluate the reported reviews. Under the assumptions that reviewers are Bayesian decision-makers and that they cannot influence the reviews of other reviewers, we show that risk-neutral reviewers strictly maximize their expected scores by honestly disclosing their reviews. We also show how the group's scores can be used to find a consensual review. Experimental results show that encouraging honest reporting through the proposed scoring method creates more accurate reviews than the traditional peer-review process.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Depth-Wise Activation Steering for Honest Language Models
Large language models sometimes assert falsehoods despite internally representing the correct answer, failures of honesty rather than accuracy, which undermines auditability and safety. Existing approaches largely optimi…
A Reputation for Honesty
We analyze situations in which players build reputations for honesty rather than for playing particular actions. A patient player facing a sequence of short-run opponents makes an announcement about their intended action…
Incentivizing honest performative predictions with proper scoring rules
Proper scoring rules incentivize experts to accurately report beliefs, assuming predictions cannot influence outcomes. We relax this assumption and investigate incentives when predictions are performative, i.e., when the…
PredictionAutomatically Detecting Self-Reported Birth Defect Outcomes on Twitter for Large-scale Epidemiological Research
In recent work, we identified and studied a small cohort of Twitter users whose pregnancies with birth defect outcomes could be observed via their publicly available tweets. Exploiting social media's large-scale potentia…
BIG-bench Machine LearningToward Honest Language Models for Deductive Reasoning
Deductive reasoning is the process of deriving conclusions strictly from the given premises, without relying on external knowledge. We define honesty in this setting as a model's ability to respond only when the conclusi…
Reinforcement Learning