The Extent of the Use of GRADE in Campbell Systematic Reviews: A Systematic Survey
Bibliographic record
Abstract
Objective: To conduct a systematic survey on the extent of the use of the grading of recommendations, assessment, development, and evaluations (GRADE) and other evidence grading systems in Campbell systematic reviews (SRs). Study Design and Settings: Campbell SRs published before January 1st, 2024, that used evidence grading systems were included. General characteristics and details of a summary of findings (SoF) table and an evidence profile (EP) were independently extracted by two investigators. Results: Among 234 SRs retrieved, 46 (19.66%) used evidence grading systems, all of which were GRADE. One used GRADE erroneously to rate the quality of individual studies rather than the body of evidence. The 45 SRs used GRADE to assess the certainty of a body of evidence and included 858 outcomes. Of them, the certainty in evidence for 32 were rated as high (3.73%), 170 were moderate (19.81%), 291 were low (33.92%), and 365 were very low (42.54%). Among the 1860 instances of downgrading and upgrading, the certainty in evidence was mostly downgraded for risk of bias (ROB) (1026, 55.16%) and imprecision (408, 21.94%). The large magnitude of effect (14, 0.75%) and plausible confounding (10, 0.54%) were the main upgraded factors. The proportions for higher certainty in evidence (including high and moderate) were larger in the international development (9.59%) and social welfare (7.55%) groups than in the other groups (1.37%). Conclusion: Most Campbell SRs do not assess the GRADE certainty in evidence. Where evidence is evaluated, the quality of that evidence is mainly low or very low, and this is most commonly due to serious ROB or imprecision.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.486 | 0.548 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.036 | 0.009 |
| Bibliometrics | 0.001 | 0.009 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.001 | 0.000 |
| Open science | 0.009 | 0.001 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; both teacher heads agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".