Do peer reviewers comment on reporting items as instructed by the journal? A secondary analysis of two randomized trials
Notice bibliographique
Résumé
OBJECTIVES: Two studies randomizing manuscripts submitted to biomedical journals have previously shown that reminding peer reviewers about key reporting items did not improve the reporting quality in published articles. Within this secondary analysis of peer reviewer reports we aimed to assess at what stage the intervention failed. STUDY DESIGN AND SETTING: We exploratively analyzed peer reviewer reports from two published randomized controlled trials (RCTs) conducted at biomedical journals. The first RCT (CONSORT-PR) assessed adherence to the Consolidated Standards of Reporting Trials (CONSORT) guideline in manuscripts presenting primary RCT results. The second RCT (SPIRIT-PR) included manuscripts presenting RCT protocols and assessed adherence to the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) guideline. In both RCTs the control group consisted of peer reviewers receiving no reminder, whereas all reviewers in the intervention group received a reminder of the 10 most important reporting items. For this secondary analysis, we extracted from peer reviewer reports which of the ten key reporting items were mentioned by reviewers as requiring clarification. The main outcome of this secondary analysis was the difference in the mean proportion of these ten reporting items for which at least one peer reviewer requested clarification. Furthermore, we assessed how this difference changed (i) if only published manuscripts were considered and (ii) when only requested changes that were implemented by authors were considered. RESULTS: We assessed peer reviewer reports from 533 manuscripts (n = 265 intervention group; n = 268 control group). Among the manuscripts in the intervention group, 21.1% (95% CI, 18.6%-23.6%) of the ten reporting items were requested for clarification, compared to 13.1% (95% CI, 18.6%-23.6%) in the control group, resulting in a mean difference of 8.0% (95% CI, 4.9%-11.1%). However, this difference diminished to 4.2% when assessing solely accepted and published manuscripts and was even further reduced to 2.6% when accounting for changes actually implemented by authors. CONCLUSION: Reminding peer reviewers to check reporting items increased their focus on reporting guidelines, leading to more reporting-related requests in their reviews. However, the effect was strongly diluted during the peer review process due to rejected articles and requests not implemented by authors. PLAIN LANGUAGE SUMMARY: When new research is submitted to a journal, other experts in the field (peer reviewers) check the research to make sure it's reliable and clear. Among others, one important part of this process is ensuring that researchers follow reporting guidelines about what information should be included in their papers so that the readers can understand how the research was conducted. We wanted to find out if reminding peer reviewers to focus on the key parts of these guidelines (ie, 10 most important items) would help to improve the reporting quality of published research papers. For this purpose, we conducted two studies in which we randomized manuscripts to either an intervention group or a control group. In the intervention group, the peer reviewers from half of the included manuscript received such a reminder (ie, asking them to check whether the 10 most important reporting items are well described in the manuscript), whereas peer reviewers in the control group did not receive a reminder. Within our previously published main results of these studies we saw that the reporting quality of the published articles did not improve with this intervention. To find out why this approach did not work, we looked closer at the individual reports from peer reviewers and checked how often reviewers asked for these important details and whether authors made the necessary changes. We found that reminders did lead to more requests about reporting items from peer reviewers. However, as a high proportion of peer-reviewed articles is rejected during the peer review process and because not all requests for improvements are addressed by authors, this effect was not visible anymore (ie, "diluted") when assessing published research articles.
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction machine sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Le volet Gemma est une étiquette directe du modèle pour chaque travail de la base, lue sur la notice réduite au titre. Le volet Codex est un classifieur appris des 10 348 étiquettes directes de Codex et calibré sur les taux pondérés de l'échantillon; les champs sans appui suffisant ne portent aucun appel Codex. Le mode candidate est l'union des deux volets; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont pas des étiquettes humaines.
Scores du classifieur distillé par catégorie (deux têtes)
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,425 | 0,788 |
| Méta-épidémiologie (sens strict) | 0,003 | 0,003 |
| Méta-épidémiologie (sens large) | 0,011 | 0,017 |
| Bibliométrie | 0,011 | 0,009 |
| Études des sciences et des technologies | 0,003 | 0,005 |
| Communication savante | 0,006 | 0,007 |
| Science ouverte | 0,004 | 0,004 |
| Intégrité de la recherche | 0,006 | 0,004 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,007 | 0,002 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; l’étiquette directe de Gemma et le classifieur distillé Codex s’accordent sur ce qui est montré ici.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».