Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective
Peer review is the primary means of quality control in academia; as an outcome of a peer review process, program and area chairs make acceptance decisions for each paper based on the review reports and scores they received. Quality of scientific work is multi-faceted; coupled with the subjectivity of reviewing, this makes final decision making difficult and time-consuming. To support this final step of peer review, we formalize it as a paper ranking problem. We introduce a novel, multi-faceted generic evaluation framework for ranking submissions based on peer reviews that takes into account effectiveness, efficiency and fairness. We propose a preference learning perspective on the task that considers both review texts and scores to alleviate the inevitable bias and noise in reviews. Our experiments on peer review data from the ACL 2018 conference demonstrate the superiority of our preference-learning-based approach over baselines and prior work, while highlighting the importance of using both review texts and scores to rank submissions.
Code (0)
등록된 구현이 없습니다.
Tasks
Decision MakingFairnessMethods 이 논문이 사용한 방법론
Similar Papers 제목 키워드 기반
Position on LLM-Assisted Peer Review: Addressing Reviewer Gap through Mentoring and Feedback
The rapid expansion of AI research has intensified the Reviewer Gap, threatening the peer-review sustainability and perpetuating a cycle of low-quality evaluations. This position paper critiques existing LLM approaches t…
Automated scholarly paper review: Concepts, technologies, and challenges
Peer review is a widely accepted mechanism for research evaluation, playing a pivotal role in academic publishing. However, criticisms have long been leveled at this mechanism, mostly because of its poor efficiency and l…
Logical ReasoningZero-shot reasoning for simulating scholarly peer-review
The scholarly publishing ecosystem faces a dual crisis of unmanageable submission volumes and unregulated AI, creating an urgent need for new governance models to safeguard scientific integrity. The traditional human-onl…
Attitudes toward Open Access, Open Peer Review, and Altmetrics among Contributors to Spanish Scholarly Journals
This paper aims to gain a better understanding of the perspectives of contributors to Spanish academic journals regarding open access, open peer review, and altmetrics. It also explores how age, gender, professional expe…
MOPRD: A multidisciplinary open peer review dataset
Open peer review is a growing trend in academic publications. Public access to peer review data can benefit both the academic and publishing communities. It also serves as a great support to studies on review comment gen…
Comment GenerationReview Generation