Recommendation to change the peer review process
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
As long-time subscribers, readers, authors, and peer-reviewers for Birth Defects Research (BDR, formerly Teratology), we believe the peer review process has been deteriorating. This is seen throughout the scientific literature and is not merely in BDR. With the proliferation of new journals internationally, the amount of inferior science published in the literature is increasing, so the peer review process must be strengthened. To this end, we propose the following two recommendations for discussion and encourage a rapid change to a new Peer Review format. We realize any changes to the peer review process must pass through several levels within the editorial office, as well as the Society for Birth Defects Research and Prevention, before being approved by the publisher, Wiley Publishing. We also recognize publishers (e.g., Wiley and Elsevier) are well aware of the subject and have collected much useful information available on their websites (Wiley Author Services, n.d.-a; Elsevier, n.d.). BDR uses a single anonymized model, as shown on the Wiley website (Wiley Author Services, n.d.-b). Readers are encouraged to search the fairly extensive literature, perhaps starting with our references listed in Supplemental Information. Wiley has collected much of this information and summarized various options, with pros and cons for each (Wiley Author Services, n.d.-c). At first, we believed a system allowing more ways to review and comment on a manuscript would lead to a better product; however, implementing such procedures likely would be cumbersome and/or expensive. Other models can be implemented almost immediately, which would greatly improve the transparency and legitimacy of BDR's peer review process. Several options are available to change the BDR's peer review system. One would be to adopt the “Transparent Peer Review” (TPR) model (Wiley Author Services, n.d.-c) in which the review is published (or posted) along with the article (authors have the option to decline TPR, and reviewers can choose whether or not to be identified). Wiley has launched a TPR program with over 60 participating journals (Wiley Author Services, n.d.-d). A link to a TPR example is located in Wiley Author Services (n.d.-d), as well as links to all other journals within the program. Additionally, as presented in the associated figure, one can read the benefits that accrue to authors, research in general, and the publisher. A study of this available model of published reviews found increased helpful reviewer comments and positive recommendations (Bravo et al., 2019; Communications Physics, 2022; Cosgrove & Cheifet, 2018). Another option is to incentivize the review process in a way that motivates reviewers to put forth a strong effort. For instance, reviewers could be offered a tangible reward for peer review (Brainard, 2021; Cheah & Piasecki, 2022). Indirect rewards (e.g., being a good academic citizen, participating in peer review as a usual part of work) are well established. If an honorarium were offered, it could be accepted, declined, donated, used to access articles, or used to help with various article processing charges. We acknowledge that there are arguments against monetary incentives. Still, a tangible reward, such as an honorarium or points that could be accrued and used to purchase Wiley books at a discount, may (1) enlarge the pool of reviewers, (2) engender motivation to review, and (3) increase the speed of reviews. It is likely that together with the publication of reviewers’ reports, there would be accountability and improved quality of published manuscripts. While the two previous suggestions would help to improve the logistical process, in the end, successful peer review comes down to individual reviewers. We do not presume to give specific instructions or recommendations on conducting a peer review, and perhaps our societies should organize reoccurring classes or workshops on the subject. In addition, the Wiley website has valuable materials for the novice peer reviewer (Wiley Author Services, n.d.-e), and Elsevier Researcher Academy offers an online certification course (Elsevier Researcher Academy, n.d.). By accepting an invitation, the reviewers promise to give the assignment their experience, wisdom, unbiased skepticism, work ethic, and willingness to give back to the scientific method. An astute reviewer will assess his/her capability to review the manuscript and/or recognize the potential for perceived bias. Upon assuming the task, a good reviewer will assess whether the experimental design is appropriate for the question at hand, methods are acceptable, data presentation is proper, and figures and photographs are clear. Peer reviewers must then decide if those methods and results lead to conclusions drawn by the authors, and if the authors have made adequate comparisons to previous/similar studies, as well as, perhaps, suggestions for future research. Ultimately, these subjective aspects must be self-taught during continuous training in the scientific method. With this said, we believe that reviewers should be proud to affix their names to their reviews, as we shall do in the future. We are not alone in recognizing the amount of poor science being published and have heard complaints from others about the current status of the peer review system. But in the final analysis, it is up to you, the readership, to take action if you want to fix the problem. This means continued, dedicated participation in the review process, and thinking about ways to improve it. We invite comments and discussion, especially to our suggestions as listed above. The authors report no conflict of interest. Data S1. Supporting information. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.436 | 0.227 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.013 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.001 | 0.000 |
| Open science | 0.003 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.031 | 0.133 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it