Search for unpublished data by systematic reviewers: an audit
Bibliographic record
Abstract
OBJECTIVES: We audited a selection of systematic reviews published in 2013 and reported on the proportion of reviews that researched for unpublished data, included unpublished data in analysis and assessed for publication bias. DESIGN: Audit of systematic reviews. DATA SOURCES: . We also searched the Cochrane Library and included 100 randomly selected Cochrane reviews. ELIGIBILITY CRITERIA: Systematic reviews published in 2013 in the selected journals were included. Methodological reviews were excluded. DATA EXTRACTION AND SYNTHESIS: Two reviewers independently reviewed each included systematic review. The following data were extracted: whether the review searched for grey literature or unpublished data, the sources searched, whether unpublished data were included in analysis, whether publication bias was assessed and whether there was evidence of publication bias. MAIN FINDINGS: 203 reviews were included for analysis. 36% (73/203) of studies did not describe any attempt to obtain unpublished studies or to search grey literature. 89% (116/130) of studies that sought unpublished data found them. 33% (68/203) of studies included an assessment of publication bias, and 40% (27/68) of these found evidence of publication bias. CONCLUSION: A significant fraction of systematic reviews included in our study did not search for unpublished data. Publication bias may be present in almost half the published systematic reviews that assessed for it. Exclusion of unpublished data may lead to biased estimates of efficacy or safety in systematic reviews.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.496 | 0.638 |
| Meta-epidemiology (narrow) | 0.012 | 0.009 |
| Meta-epidemiology (broad) | 0.029 | 0.022 |
| Bibliometrics | 0.087 | 0.072 |
| Science and technology studies | 0.006 | 0.009 |
| Scholarly communication | 0.010 | 0.016 |
| Open science | 0.011 | 0.012 |
| Research integrity | 0.011 | 0.010 |
| Insufficient payload (model declined to judge) | 0.024 | 0.015 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".