Program evaluation with multilevel longitudinal data: evidence from simulation study and cluster randomized controlled trial
Bibliographic record
Abstract
Background: Many intervention programs are implemented with cluster randomized controlled trial (cRCT), i.e., the clusters (e.g., classrooms or schools), not subjects, were randomly assigned into treatment or control groups. The outcome variables are also reported for multiple time points (e.g., pre-and post-intervention). The mixed model is commonly used in analyzing longitudinal data, and most research on program evaluation ignored the non-independence of subjects within-cluster even data are from cluster sampling design. Ignoring the dependency between measurements at different times within-subject has been shown that it can lead to the incorrect estimates of standard error and the type-I-error, but the consequence of ignoring non-independence of subjects within-cluster and/or between measurements within-subject has not been investigated extensively. Objectives: The objectives of this study are, (i) to examine the impact of ignoring the within-cluster correlation and/or within-subject correlations on program evaluation in the cRCT studies; (ii) to evaluate the effect of a mental health prevention program with the cRCT design and investigate factors that moderate the successful intervention. Methods: We implemented both simulation and application to real study to illustrate the impact of ignoring non-independence on effect size estimation in the cRCT. Project 11, a prevention program in Manitoba schools to improve mental health, was used as an illustration example for the empirical study. This study has been implemented with cRCT by randomizing the classrooms into treatment or control groups. Three-time repeated measurements of each student clustered within classroom exhibit a three-level hierarchy of data structure. Based on this data, we simulated three-level data with different magnitudes of intraclass correlation to represent different degrees to which individuals resemble each other relatedness within the cluster. We applied both 2-level (ignoring a level, i.e., either within-class correlation or within-subject correlation) and 3-level (considering both correlation terms) multilevel models to compare the outcome of interest with the true population parameters. The Project 11 data was used as an example to illustrate the consequence of ignoring the higher level of hierarchy on the estimation of intervention and moderation effects. Results: The simulation study shows that ignoring the within-cluster correlation and/or within-subject correlations gives less accurate parameter estimates, and the coverage rate also decreases. ii This impact depends on the sample size and ICC of each level of the multilevel data, and for small sample size, the impact is found severe. The results of the empirical data analysis show that both the random effect and fixed effect parameter estimates along with their standard errors get affected if a level is ignored. The analysis of Project 11 data provides evidence of the positive effect of this cRCT based mental health intervention program. The behavioural difficulties of students significantly decrease over time, and socioeconomic status (SES) has a moderation effect on the program outcome. Although gender does not moderate the effect of the intervention program directly, significant gender difference on the moderation effect of SES is observed. Conclusions: In cRCT based study, it is important to consider the within-cluster correlation and/or within-subject correlations as ignoring these correlations gives incorrect results and, therefore, can lead to different research conclusions. Project 11 program effectively reduces participated students’ behavioural difficulties, and SES significantly moderates the outcome. The study provides guidance for school-based program design and evaluation, and we can learn more about how and for whom interventions work.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.177 | 0.460 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.007 | 0.013 |
| Bibliometrics | 0.003 | 0.004 |
| Science and technology studies | 0.001 | 0.003 |
| Scholarly communication | 0.003 | 0.004 |
| Open science | 0.004 | 0.003 |
| Research integrity | 0.004 | 0.005 |
| Insufficient payload (model declined to judge) | 0.008 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".