No Agreement Without Loss: Learning and Social Choice in Peer Review
In peer review systems, reviewers are often asked to evaluate various features of submissions, such as technical quality or novelty. A score is given to each of the predefined features and based on these the reviewer has to provide an overall quantitative recommendation. It may be assumed that each reviewer has her own mapping from the set of features to a recommendation, and that different reviewers have different mappings in mind. This introduces an element of arbitrariness known as commensuration bias. In this paper we discuss a framework, introduced by Noothigattu, Shah and Procaccia, and then applied by the organizers of the AAAI 2022 conference. Noothigattu, Shah and Procaccia proposed to aggregate reviewer's mapping by minimizing certain loss functions, and studied axiomatic properties of this approach, in the sense of social choice theory. We challenge several of the results and assumptions used in their work and report a number of negative results. On the one hand, we study a trade-off between some of the axioms proposed and the ability of the method to properly capture agreements of the majority of reviewers. On the other hand, we show that dropping a certain unrealistic assumption has dramatic effects, including causing the method to be discontinuous.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
Loss Functions, Axioms, and Peer Review
It is common to see a handful of reviewers reject a highly novel paper, because they view, say, extensive experiments as far more important than novelty, whereas the community as a whole would have embraced the paper. Mo…
Keep it green, simple and socially fair: a choice experiment on prosumers' preferences for peer to peer electricity trading in the Netherlands
While the potential for peer-to-peer electricity trading, where households trade surplus electricity with peers in a local energy market, is rapidly growing, the drivers of participation in this trading scheme have been …
energy tradingMarketingSocial Learning with Intrinsic Preferences
Despite strong evidence for peer effects, little is known about how individuals balance intrinsic preferences and social learning in different choice environments. Using a combination of experiments and discrete choice m…
Mapping Social Choice Theory to RLHF
Recent work on the limitations of using reinforcement learning from human feedback (RLHF) to incorporate human preferences into model behavior often raises social choice theory as a reference point. Social choice theory'…
reinforcement-learningModelling Social Evolutionary Processes and Peer Effects in Agricultural Trade Networks: the Rubber Value Chain in Indonesia
Understanding market participants' channel choices is important to policy makers because it yields information on which channels are effective in transmitting information. These channel choices are the result of a recurs…