Replication Package and Online Appendix for "Characterizing the prevalence, distribution, and duration of stale reviewer recommendations"
Bibliographic record
Abstract
Repository Overview This script will download everything you need to replicate our study into your computer. Welcome to the repository for our research study titled "Characterizing the Impact, Distribution, and Duration of Stale Reviewer Recommendations." This package is designed to assist researchers and practitioners in replicating our study by providing the necessary code and dataset, along with all the materials required to reproduce our analyses, tables, and figures. Contents The repository contains two main folders: docker_images.zip: This compressed file includes the docker images built for this replication package. If you have access to dockerhub on your machine, you may not need to download it. Please refer to README.md and INSTALL.md for further information. INSTALL.MD: This markdown file contains help setting up the system. README.MD: This file contains the guide on how to run the scripts and the content of this replication package. script.zip: Contains the scripts required to run this study's experiments. replication.zip: This file contains the dataset and code used in our study. By accessing this package, you can replicate our research and examine the results of our experiments. To get started, please unzip the file and follow the instructions in the README.md file to access the dataset and replicate our study successfully. Online Appendix: In this folder, you will find the online appendix that complements our main research. Feel free to explore this section to gain additional insights and information related to our study. Need Help? If you have any concerns or questions during the replication process or while working with our materials, please don't hesitate to contact us. We are more than happy to assist you and ensure a smooth experience. I appreciate your interest in our research! We hope you find this repository valuable for your work and studies.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.068 | 0.442 |
| Meta-epidemiology (narrow) | 0.003 | 0.003 |
| Meta-epidemiology (broad) | 0.003 | 0.004 |
| Bibliometrics | 0.008 | 0.010 |
| Science and technology studies | 0.005 | 0.002 |
| Scholarly communication | 0.007 | 0.007 |
| Open science | 0.004 | 0.005 |
| Research integrity | 0.003 | 0.005 |
| Insufficient payload (model declined to judge) | 0.673 | 0.425 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".