Investigating the impact of trial retractions on the healthcare evidence ecosystem (VITALITY Study I): retrospective cohort study
Bibliographic record
Abstract
OBJECTIVE: To investigate the impact of retracted trials on the production and use of healthcare evidence in the evidence ecosystem. DESIGN: Retrospective cohort study based on forward citation searching. DATA SOURCES: Retraction Watch up to 5 November 2024. STUDY SELECTION: Randomised controlled trials in humans that were retracted for any reason. METHODS: Forward citation searching via Google Scholar and Scopus was used to identify evidence synthesis research (21 November 2024) that quantitatively incorporated retracted trials. Data were independently extracted by two groups of researchers. The results of meta-analyses were updated after exclusion of the retracted trials. The proportions of meta-analyses that changed direction of the pooled effect and/or the significance of the P value were estimated. A generalised linear mixed model was used to investigate the association between the number of included studies and the impact, measured by odds ratio and 95% confidence interval (CI). The impact of distorted evidence on clinical practice guidelines was also investigated on the basis of citation searching. RESULTS: The searches identified 1330 retracted trials and 847 systematic reviews that quantitatively synthesised retracted trials, with a total of 3902 meta-analyses that could be replicated. After the potential clustering effects were accounted for, the exclusion of the retracted trials led to a change in the direction of the pooled effect in 8.4% (95% CI 6.8% to 10.1%), in its statistical significance in 16.0% (14.2% to 17.9%), and in both direction and significance in 3.9% (2.5% to 5.2%) and a >50% change in the magnitude of the effect in 15.7% (13.5% to 17.9%). An obvious non-linear association existed between the number of included studies and the impact on the results, with a lower number of studies having higher impact (eg, for 10 studies versus ≥20 studies, change of direction: odds ratio 2.63, 95% CI 1.29 to 5.38; P<0.001). Evidence from 68 systematic reviews with conclusions distorted by retracted trials was used in 157 guideline documents. CONCLUSION: Retracted trials have a substantial impact on the evidence ecosystem, including evidence synthesis, clinical practice guidelines, and evidence based clinical practice. Evidence generators, synthesisers, and users must pay attention to this problem, and feasible approaches that assist with easier identification and correction of such potential contamination are needed. STUDY REGISTRATION: Open Science Framework (https://osf.io/7eazq/).
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.019 | 0.017 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.002 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".