The Systematic Removal of Participants Post-Randomization Can Lead to Alternate Explanations of the Results
Bibliographic record
Abstract
The strength of randomized controlled trials is that they allow causal statements to be made about the efficacy of an intervention. This is because the randomization of participants to experimental condition distributes participants with different characteristics to each experimental condition by chance, including variations in the outcome variables of interest. Such randomization allows the use of statistical tests incorporating probability statements regarding the chance that the difference observed between conditions is due to chance (eg, 1 in 20 chance; P<.05). Any systematic removal of participants post-randomization interferes with the assumption of the randomness of participant allocation to experimental condition. This can introduce a potential confound or alternative explanation of the results. The recently published study by Rooke et al [1] removed participants from the analysis who reported other treatment use during the study (n=5). The removal of participants was done post-randomization and in a systematic fashion (ie, anyone who reported using other treatment was not included in the analysis). All participants who reported receiving other treatment were in the control condition and were allocated to receive the Web address of an education only website. Even if some participants in the intervention condition had also accessed other treatment, this would still be a systematic, post-randomization removal of participants. However, it is easier to develop alternate explanations of the findings of this trial because all participants using other treatment were from the control condition. From one perspective, it is possible that the excluded participants were those who were experiencing the most serious problems with their Cannabis use. If this is the case, it could make it less likely that significant differences would be observed between experimental condition by reducing the variance between participants. Alternatively, perhaps these were the five participants in the control condition who were the most motivated to reduce their Cannabis use. This could mean that participants in the control condition were less motivated, on average, than those in the intervention condition, to do something about their Cannabis use. This could serve as an alternate explanation of the findings in this study. Given that only five participants were removed out of 225 (or 230 if control and intervention group participant totals are added?) randomized at baseline, it is quite possible that this alternate explanation is untrue. However, the authors do expose themselves to alternate explanations of the findings by the removal of these participants. If the systematic removal of participants post-randomization is deemed necessary, one possible solution would be to run (and report on) the analysis with and without these participants included. Such sensitivity analyses would go a long way towards addressing any possible confounds that may have been introduced.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.413 | 0.615 |
| Meta-epidemiology (narrow) | 0.005 | 0.004 |
| Meta-epidemiology (broad) | 0.012 | 0.008 |
| Bibliometrics | 0.007 | 0.007 |
| Science and technology studies | 0.003 | 0.011 |
| Scholarly communication | 0.005 | 0.007 |
| Open science | 0.005 | 0.005 |
| Research integrity | 0.012 | 0.011 |
| Insufficient payload (model declined to judge) | 0.013 | 0.006 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".