Dual Anonymous and Distributed Peer Review for Proposal Review Rankings at the ALMA Observatory

John Carpenter,1 Andrea Corvillón1 Objective In 2021,1 the Atacama Large Millimeter/submillimeter Array (ALMA) transitioned from single-anonymous panel reviews to dual-anonymous distributed peer review to manage the growing volume of proposal submissions and mitigate potential biases.2 We conducted a retrospective cohort study to examine associations between this procedural change and proposal rankings across principal investigator (PI) demographic characteristics, using 7 years of data under the previous format (2012-2018) and 4 years under the new process (2021-2024). Design We analyzed proposal rankings from 12 ALMA cycles. From 2011 to 2018 (cycles 0-6), proposals were reviewed in topical panels under single-anonymous peer review. In 2019 (cycle 7), investigator lists were randomized while panels were retained. No review process was held in 2020 due to the COVID-19 pandemic. In 2021 (cycle 8), ALMA implemented dual-anonymous, distributed peer review for most proposals. We examined rankings by 3 PI demographic characteristics: (1) experience (number of cycles in which the PI submitted proposals), (2) regional affiliation (Chile, East Asia, Europe, North America, or other), and (3) sex. We grouped proposals by review era: single-anonymous panel review (cycles 1-6; 2012-2018; 9091 proposals) and dual-anonymous distributed peer review (cycles 8-11; 2021-2024; 6490 proposals). Cycle 0 and cycle 7 were excluded from the analysis: cycle 0 because all PIs were, by definition, first-time users of ALMA, and cycle 7 because it was a transitional year for implementing dual anonymity. Results Proposal rankings were normalized from 0 (best) to 1 (worst) for comparability across cycles. Proposal counts by demographic subgroup and review era are reported in Table 25-1053. We compared median normalized rankings between review eras using 10,000 bootstrap samples to generate 95% CIs and 2-sided P values. We observed the following associations: (1) for experience, PIs submitting in all cycles had better rankings during single-anonymous panel review than under dual-anonymous distributed review (P = .006), and first-time PIs ranked lowest in both systems with no significant change in rankings (P = .19); (2) for regional affiliation, East Asian PIs showed improved rankings after the transition (P < .001), rankings for European PIs declined (P = .009) but remained above average, and rankings for PIs from North America, Chile, and other regions did not show significant changes (P > .60); and (3) for sex, no statistically significant differences in rankings were observed between male- and female-led proposals in either review system (P = .12). https://assets.underline.io/markdown_image/1/image/ef41fffb9d8404953c23a5d7853259dc.png Conclusions Dual-anonymous, distributed peer review was associated with reduced disparities by PI experience and region, consistent with reduced prestige and geographic bias. While increased score variability may have contributed, the nonuniform changes (ie, some groups improved, others remained stable) are inconsistent with a purely noise-driven explanation. These findings suggest that systematic shifts in reviewer behavior, not merely increased randomness, underlie the observed trends. Disentangling the effects of dual anonymity, distributed review, and other concurrent changes (eg, increased use of artificial intelligence) remains an important direction for future research. References 1. Donovan Meyer J, Corvillón A, Carpenter JM, et al. Analysis of the ALMA Cycle 8 distributed peer review process. Bull Am Astron Soc. 2022;54(1):43. doi:10.3847/25c2cfeb.4ece85d4 2. Carpenter JM, Corvillón A, Donovan Meyer J, et al. Update on the systematics in the ALMA proposal review process after Cycle 8. Publ Astron Soc Pac. 2022;134:045001. doi:10.1088/1538-3873/ac5b89 1Joint ALMA Observatory, Santiago, Chile, john.carpenter@alma.cl. Conflict of Interest Disclosures John Carpenter and Andrea Corvillón are employed by the Joint ALMA Observatory, which is jointly managed by Associated Universities Inc/National Radio Astronomy Observatory, the European Organisation for Astronomical Research in the Southern Hemisphere, and the National Astronomical Observatory of Japan on behalf of the ALMA partnership. Funding/Support This work was supported by the Joint ALMA Observatory. Role of the Funder/Sponsor The funder had a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and decision to submit the abstract for presentation. Acknowledgment ALMA is a partnership of the European Organization for Astronomical Research in the Southern Hemisphere (representing its member states), National Science Foundation (US), and National Institutes of Natural Sciences (Japan), together with the National Research Council of Canada (Canada), the National Science and Technology Council and Academia Sinica Institute of Astronomy and Astrophysics (Taiwan), and the Korea Astronomy and Space Science Institute (Republic of Korea), in cooperation with the Republic of Chile.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.