Assessing fidelity to family-based treatment: an exploratory examination of expert, therapist, parent, and peer ratings
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
INTRODUCTION: Fidelity is an essential component for evaluating the clinical and implementation outcomes related to delivery of evidence-based practices (EBPs). Effective measurement of fidelity requires clinical buy-in, and as such, requires a process that is not burdensome for clinicians and managers. As part of a larger implementation study, we examined fidelity to Family-Based Treatment (FBT) measured by several different raters including an expert, a peer, therapists themselves, and parents, with a goal of determining a pragmatic, reliable and efficient method to capture treatment fidelity to FBT. METHODS: Each therapist audio-recorded at least one FBT case and submitted recordings from session 1, 2, and 3 from phase 1, plus one additional session from phase 1, two sessions from phase 2, and one session from phase 3. These submitted files were rated by an expert and a peer rater using a validated FBT fidelity measure. As well, therapists and parents rated fidelity immediately following each session and submitted ratings to the research team. Inter-observer reliability was calculated for each item using the intraclass correlation coefficient (ICC), comparing the expert ratings to ratings from each of the other raters (parents, therapists, and peer). Mean scale scores were compared using repeated measures ANOVA. RESULTS: Intraclass correlation coefficients revealed that agreement was the best between expert and peer, with excellent, good, or fair agreement in 7 of 13 items from session 1, 2 and 3. There were only four such values when comparing expert to parent agreement, and two such values comparing expert to therapist ratings. The rest of the ICC values indicated poor agreement. Scale level analysis indicated that expert fidelity ratings for phase 1 treatment sessions scores were significantly higher than the peer ratings and, that parent fidelity ratings tended to be significantly higher than the other raters across all three treatment phases. There were no significant differences between expert and therapist mean scores. CONCLUSIONS: There may be challenges inherent in parents rating fidelity accurately. Peer rating or therapist self-rating may be considered pragmatic, efficient, and reliable approaches to fidelity assessment for real-world clinical settings.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.003 | 0.002 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it