How do Authors' Perceptions of their Papers Compare with Co-authors' Perceptions and Peer-review Decisions?
How do author perceptions match up to the outcomes of the peer-review process and perceptions of others? In a top-tier computer science conference (NeurIPS 2021) with more than 23,000 submitting authors and 9,000 submitted papers, we survey the authors on three questions: (i) their predicted probability of acceptance for each of their papers, (ii) their perceived ranking of their own papers based on scientific contribution, and (iii) the change in their perception about their own papers after seeing the reviews. The salient results are: (1) Authors have roughly a three-fold overestimate of the acceptance probability of their papers: The median prediction is 70% for an approximately 25% acceptance rate. (2) Female authors exhibit a marginally higher (statistically significant) miscalibration than male authors; predictions of authors invited to serve as meta-reviewers or reviewers are similarly calibrated, but better than authors who were not invited to review. (3) Authors' relative ranking of scientific contribution of two submissions they made generally agree (93%) with their predicted acceptance probabilities, but there is a notable 7% responses where authors think their better paper will face a worse outcome. (4) The author-provided rankings disagreed with the peer-review decisions about a third of the time; when co-authors ranked their jointly authored papers, co-authors disagreed at a similar rate -- about a third of the time. (5) At least 30% of respondents of both accepted and rejected papers said that their perception of their own paper improved after the review process. The stakeholders in peer review should take these findings into account in setting their expectations from peer review.
Code (0)
등록된 구현이 없습니다.
Similar Papers 제목 키워드 기반
CITEX: A new citation index to measure the relative importance of authors and papers in scientific publications
Evaluating the performance of researchers and measuring the impact of papers written by scientists is the main objective of citation analysis. Various indices and metrics have been proposed for this. In this paper, we pr…
From Paper to Card: Transforming Design Implications with Generative AI
Communicating design implications is common within the HCI community when publishing academic papers, yet these papers are rarely read and used by designers. One solution is to use design cards as a form of translational…
To Trust or Not to Trust: Authors' Response to AI-based Reviews
Large language models are increasingly discussed and used as tools that may assist with scholarly peer review, but empirical evidence regarding how authors use and perceive AI-based feedback remains limited. This paper r…
Two Decades of Network Science as seen through the co-authorship network of network scientists
Complex networks have attracted a great deal of research interest in the last two decades since Watts & Strogatz, Barab\'asi & Albert and Girvan & Newman published their highly-cited seminal papers on small-world network…
Data VisualizationDe-anonymization of authors through arXiv submissions during double-blind review
In this paper, we investigate the effects of releasing arXiv preprints of papers that are undergoing a double-blind review process. In particular, we ask the following research question: What is the relation between de-a…