Assessing the extent and quality of documentation of fidelity to the implementation strategy: a proposed scoring mechanism
Bibliographic record
Abstract
Implementation fidelity is critical to the internal and external validity of a study. Without it, accurate conclusions about an intervention cannot be drawn as unknown factors may have influenced the outcome(s). This scoping review assesses the extent and quality of documentation of fidelity in implementation studies. Articles in the Cochrane Database of Systematic Reviews, Effective Practice and Organization of Care (EPOC) that included at least one of the 47 Cochrane Review implementation strategies were retrieved yielding 1158 articles in 337 journals. We selected publications from the 3 journals with the largest number of articles: British Medical Journal (BMJ), Journal of the American Medical Association (JAMA), and Medical Care. We developed an "Implementation Strategy Fidelity Score" based on definitions adapted from Dusenbury (2003) [ 1 ]. Adherence (documenting implementation strategies and extent to which they took place), dose (proportion of people receiving each strategy) and participant responsiveness (extent to which individuals were involved in the development or evaluation of strategies or their receptivity to them) were scored on a 3-point scale: 0=no detail, 1=some detail, and 2=great detail. The number of articles included for each journal and mean scores (standard deviations) across categories were: BMJ: n = 28; adherence = 0.86 (SD = 0.87); dose = 0.57 (SD = 0.86); participant responsiveness = 0.79 (SD = 0.90). JAMA: n = 26: adherence = 0.88 (SD = 0.95); dose = 0.76 (SD = 0.81); participant responsiveness = 0.92 (SD = 0.93). Medical Care: n = 19; adherence = 1.11 (SD = 0.97); dose = 0.95 (SD = 1.0); participant responsiveness = 1.21 (SD = 0.89). Although mean scores for the Medical Care journal were higher in each category, documentation of fidelity to the implementation strategy overall can be improved by adopting standard definitions, and better practices for measurement, documentation and reporting. This scoring mechanism provides insight into the documentation of fidelity. It also provides a structure for defining and documenting fidelity to the implementation strategy and a benchmark from which to measure the quality of documentation.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.508 | 0.590 |
| Meta-epidemiology (narrow) | 0.002 | 0.002 |
| Meta-epidemiology (broad) | 0.004 | 0.008 |
| Bibliometrics | 0.028 | 0.019 |
| Science and technology studies | 0.003 | 0.007 |
| Scholarly communication | 0.007 | 0.010 |
| Open science | 0.006 | 0.012 |
| Research integrity | 0.003 | 0.003 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".