8 When should systematic reviews be replicated and when is it wasteful: a checklist and framework
Bibliographic record
Abstract
In my postdoctoral research, I have used an evidence-driven, transparent process and implemented consensus approaches to develop value-added guidance on when and how to replicate systematic reviews. As outlined below, the aims, methodological approach and dissemination strategies of this project align closely with the EBM manifesto of making evidence relevant, replicable, and accessible to end-users. Background Replication is a cornerstone of the scientific method, yet replication of systematic reviews is too often overlooked, done unnecessarily or done poorly. Systematic review replication is conducted with the objective of testing whether results of an index review can be repeated or extended. Failure to replicate may lead to continued uncertainty about the implications of a body of evidence. The compelling case for replicating systematic reviews is complicated by concerns about research waste – too frequent replication of systematic reviews can represent an inefficient use of scarce research resources. There is a lack of guidance for when to, and when not to replicate systematic reviews. Objective To develop evidence-driven, consensus-based recommendations on when and how to replicate systematic reviews, taking into account the needs and preferences of the various stakeholder groups. METHODS: We used an integrated knowledge translation approach by involving an international multidisciplinary team of methodologists and knowledge users (authors, commissioners, funders, and consumers of systematic reviews, including patients, clinicians, and representatives from organizations involved with policy-making) at every stage of this research. The project was conducted in 4 phases: 1) semi-structured interviews with key informants to seek their opinions on definitions and criteria for systematic review replication; 2) a systematic review of evidence on when and how to replicate systematic reviews and an analysis of discordant reviews; 3) an online survey of knowledge users to assess level of agreement on draft criteria for systematic review replication; 4) a consensus meeting of 36 participants representing key stakeholder groups: patients, clinicians, journal editors, researchers, systematic review organizations, and guideline developers, to discuss the findings of the first three phases of the project and seek agreement on a checklist and framework for systematic review replication. Results Based on the opinion-gathering, literature review, and consensus meeting discussions, we developed: 1) a 4-item checklist applying the value of information (VOI) concept to determine whether the benefits of replicating an existing systematic review outweigh alternative uses of resources; and 2) a framework to determine how issues in the conduct of an index review represent threats to validity sufficient to justify formal replication and what are the appropriate review methods to address the specific threat of validity within the replicated systematic review. Conclusions Given the role of systematic reviews in policy-making and guideline development, the validity and reliability of their findings should be tested. The checklist and framework serve as explicit prompts to carefully consider the value of systematic review replication. Next steps will include assessing usability and acceptability of the checklist and framework, and adapting them to different users.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.767 | 0.773 |
| Meta-epidemiology (narrow) | 0.007 | 0.008 |
| Meta-epidemiology (broad) | 0.010 | 0.015 |
| Bibliometrics | 0.032 | 0.022 |
| Science and technology studies | 0.018 | 0.042 |
| Scholarly communication | 0.040 | 0.044 |
| Open science | 0.027 | 0.032 |
| Research integrity | 0.033 | 0.033 |
| Insufficient payload (model declined to judge) | 0.005 | 0.005 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".