Anonymizing Reviewers to Each Other in Peer Review Discussions: A Randomized Controlled Trial

Charvi Rastogi,¹ Xiangchen Song,² Zhijing Jin,^3,4 Ivan Stelmakh,⁵ Hal Daumé III,⁶ Kun Zhang,² Nihar B. Shah² <h4>Objective </h4> Many peer-review processes in computer science involve reviewers submitting independent reviews followed by a discussion between reviewers of each article on a typed forum (online discussion board). A key policy question is whether reviewers should remain anonymous to each other. This study investigated 7 research questions (RQs): RQ 1. Do reviewers discuss more when anonymous to each other or not? RQ 2. Are decisions closer to senior or junior reviewers’ opinions across conditions? RQ 3. Are reviewers more polite when not anonymous? RQ 4. Do self-reported reviewer experiences differ? RQ 5. Do reviewers prefer one condition? RQ 6. What factors do reviewers consider important in this policy decision? RQ 7. Have reviewers experienced dishonest behavior when their identity is revealed to other reviewers? <h4>Design </h4> A randomized controlled trial was conducted in the Conference on Uncertainty in Artificial Intelligence (UAI) in 2022, where full articles (not abstracts) were reviewed. Reviewers and articles were randomly assigned to either a condition where reviewer identities were hidden from each other or one where they were visible. Reviewers were then matched to articles within each condition using a semiautomated procedure.¹ An anonymous survey of reviewers was also administered. The following measurements were made: RQ 1. Average posts per reviewer-article pairs were compared; test statistic: difference across conditions. RQ 2. Test statistic: difference in the fraction of articles where the reviewer closest to the final decision was senior. RQ 3. Politeness was scored 1 to 5 using a locally deployed large language model² with few-shot prompting; scores were averaged across iterations and paraphrased prompts; test statistic: normalized Mann-Whitney U test. RQ 4. Reviewers rated 5 aspects of their experience on a 5-point Likert scale; differences across conditions were tested using a normalized Mann-Whitney U test. RQ 5. Reviewers rated overall preference on a 5-point scale mapped from −2 to 2; test statistic: Cohen <i>d</i>. RQ 6. Reviewers rated the importance of 6 factors in deciding on anonymity policy, each from 1 (least important) to 6. RQ 7. Reviewers reported experience of any dishonest behavior due to reviewer identities being visible to other reviewers, with checkboxes “Yes, in UAI 2022,” “Yes, in another venue,” “Not sure,” and “No.” The test statistics also served as a measure of the effect sizes. <i>P</i> values were computed via permutation testing. <h4>Results </h4> Overall, 322 papers were reviewed under the anonymous condition (116 accepted) and 310 papers under nonanonymous (114 accepted), with exactly 289 reviewers in both conditions. There were 611 discussion posts made by reviewers in the anonymous condition and 514 in the nonanonymous condition. The results for the 7 research questions are provided in <b>Table 24-0812</b>. https://assets.underline.io/markdown_image/1/image/bd5b7dcf2758e3af6ff9169164e2cab0.png <h4>Conclusions </h4> Small but significant differences favoring anonymous discussions were found. Subsequent computer science conferences have drawn on these findings for their policy choices, with a greater inclination toward anonymity in reviewer discussions. <h4>References</h4> 1. Shah N. An overview of challenges, experiments, and computational solutions in peer review (extended version). July 7, 2025. Accessed July 16, 2025. <a href="https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf"><span class="CharOverride-10">https://www.cs.cmu.edu/~nihars/preprints/SurveyPeerReview.pdf</span></a> 2. Chiang W, Li Z, Lin Z, et al. Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. LMSYSORG. March 2023. <a href="https://lmsys.org/blog/2023-03-30-vicuna/"><span class="CharOverride-10">https://lmsys.org/blog/2023-03-30-vicuna/</span></a> ¹Google DeepMind, New York, NY, US; ²Carnegie Mellon University, Pittsburgh, PA, US, nihars@cs.cmu.edu; ³ETH Zurich, Zurich, Switzerland; ⁴Max Planck Institute, Tübingen, Germany; ⁵New Economic School, Moscow, Russia; ⁶University of Maryland, Baltimore, MD, US. <h4>Conflict of Interest Disclosures</h4> As per author affiliations above. In addition, Zhijing Jin is going to join the University of Toronto, and Kun Zhang has a partial appointment at Mohamed bin Zayed University of Artificial Intelligence. <h4>Funding/Support</h4> ONR N000142212181, NSF 1942124, 2200410, 2229881, NIH R01HL159805. <h4>Role of Funder/Sponsor</h4> The funders played no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; and decision to submit the abstract for presentation. <h4>Acknowledgment</h4> This work was conducted when Charvi Rastogi and Ivan St

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.