Do process evaluations open up the ‘black box’ of implementation interventions in health care? A scoping review
Bibliographic record
Abstract
BACKGROUND: Process evaluations are considered an essential component in conducting and reporting complex interventions, such as those studied in randomised controlled trials (RCTs) of implementation interventions, to explain the effect of implementation interventions. Given the growth of RCTs of implementation interventions with embedded process evaluations, it is timely to review the explanatory learnings to date. This scoping review explores process evaluations of RCTs of implementation interventions to examine how studies are conducted and what insights can be offered about how and why implementation interventions achieve (or not) their intended impacts. METHODS: The scoping review was conducted in accordance with the JBI methodology. MEDLINE, CINAHL, Scopus, Web of Science and PsycINFO were searched. Articles were screened and data were extracted by two independent reviewers. RESULTS: Of the 5857 studies screened, 81 process evaluations were included. Two process evaluations reported on the same trial, resulting in a final number of n = 80 independent studies. Half of studies (48%) reported on implementation trials with no demonstrated effect on the primary outcome (null), while n = 32 (40%) reported on trials where the intervention group demonstrated positive changes in the primary outcome (positive). Seven studies (9%) had mixed findings and n = 3 (4%) studies had no reported trial outcomes. When comparing process evaluation findings from positive and null trials, few discernible patterns that clearly explained the difference in outcomes were identified. Education and training was the most common strategy used in implementation interventions, yet one of the most common implementation barriers reported related to knowledge and self-efficacy, which could indicate a misalignment. Availability of resources was the most prominent barrier for both positive and null trials and there was little evidence that implementation interventions were tailored to context despite prominent barriers and enablers at the inner and outer setting level. CONCLUSIONS: Process evaluation studies embedded in RCTs of implementation interventions are recommended as an important method to explain whether and how interventions produce their intended effect. This review suggests a need to further optimise the design and evaluation of implementation interventions, including the conduct and reporting of process evaluations, to continue advancing the science and practice of implementation. TRIAL REGISTRATION: Protocol published in Open Science Framework, May 10 2022 (Collyer et al., Process evaluations in randomised trials of implementation interventions in health care: a scoping review protocol. In Open science framework, 2022).
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.356 | 0.663 |
| Meta-epidemiology (narrow) | 0.003 | 0.004 |
| Meta-epidemiology (broad) | 0.011 | 0.011 |
| Bibliometrics | 0.033 | 0.038 |
| Science and technology studies | 0.004 | 0.010 |
| Scholarly communication | 0.022 | 0.032 |
| Open science | 0.006 | 0.009 |
| Research integrity | 0.010 | 0.007 |
| Insufficient payload (model declined to judge) | 0.005 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".