Program evaluation with multilevel longitudinal data: evidence from simulation study and cluster randomized controlled trial
Notice bibliographique
Résumé
Background: Many intervention programs are implemented with cluster randomized controlled trial (cRCT), i.e., the clusters (e.g., classrooms or schools), not subjects, were randomly assigned into treatment or control groups. The outcome variables are also reported for multiple time points (e.g., pre-and post-intervention). The mixed model is commonly used in analyzing longitudinal data, and most research on program evaluation ignored the non-independence of subjects within-cluster even data are from cluster sampling design. Ignoring the dependency between measurements at different times within-subject has been shown that it can lead to the incorrect estimates of standard error and the type-I-error, but the consequence of ignoring non-independence of subjects within-cluster and/or between measurements within-subject has not been investigated extensively. Objectives: The objectives of this study are, (i) to examine the impact of ignoring the within-cluster correlation and/or within-subject correlations on program evaluation in the cRCT studies; (ii) to evaluate the effect of a mental health prevention program with the cRCT design and investigate factors that moderate the successful intervention. Methods: We implemented both simulation and application to real study to illustrate the impact of ignoring non-independence on effect size estimation in the cRCT. Project 11, a prevention program in Manitoba schools to improve mental health, was used as an illustration example for the empirical study. This study has been implemented with cRCT by randomizing the classrooms into treatment or control groups. Three-time repeated measurements of each student clustered within classroom exhibit a three-level hierarchy of data structure. Based on this data, we simulated three-level data with different magnitudes of intraclass correlation to represent different degrees to which individuals resemble each other relatedness within the cluster. We applied both 2-level (ignoring a level, i.e., either within-class correlation or within-subject correlation) and 3-level (considering both correlation terms) multilevel models to compare the outcome of interest with the true population parameters. The Project 11 data was used as an example to illustrate the consequence of ignoring the higher level of hierarchy on the estimation of intervention and moderation effects. Results: The simulation study shows that ignoring the within-cluster correlation and/or within-subject correlations gives less accurate parameter estimates, and the coverage rate also decreases. ii This impact depends on the sample size and ICC of each level of the multilevel data, and for small sample size, the impact is found severe. The results of the empirical data analysis show that both the random effect and fixed effect parameter estimates along with their standard errors get affected if a level is ignored. The analysis of Project 11 data provides evidence of the positive effect of this cRCT based mental health intervention program. The behavioural difficulties of students significantly decrease over time, and socioeconomic status (SES) has a moderation effect on the program outcome. Although gender does not moderate the effect of the intervention program directly, significant gender difference on the moderation effect of SES is observed. Conclusions: In cRCT based study, it is important to consider the within-cluster correlation and/or within-subject correlations as ignoring these correlations gives incorrect results and, therefore, can lead to different research conclusions. Project 11 program effectively reduces participated students’ behavioural difficulties, and SES significantly moderates the outcome. The study provides guidance for school-based program design and evaluation, and we can learn more about how and for whom interventions work.
Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.
Comment cette classification a été obtenuedéplier
Prédiction machine sur la base complète
Imitation des enseignantsNi prévalence calibrée, ni vérité terrain. Validation humaine à venir. Le volet Gemma est une étiquette directe du modèle pour chaque travail de la base, lue sur la notice réduite au titre. Le volet Codex est un classifieur appris des 10 348 étiquettes directes de Codex et calibré sur les taux pondérés de l'échantillon; les champs sans appui suffisant ne portent aucun appel Codex. Le mode candidate est l'union des deux volets; le consensus est leur intersection. Ces sorties portent le statut machine_predicted_unvalidated et ne sont pas des étiquettes humaines.
Scores du classifieur distillé par catégorie (deux têtes)
| Catégorie | Codex | Gemma |
|---|---|---|
| Métarecherche | 0,177 | 0,460 |
| Méta-épidémiologie (sens strict) | 0,002 | 0,001 |
| Méta-épidémiologie (sens large) | 0,007 | 0,013 |
| Bibliométrie | 0,003 | 0,004 |
| Études des sciences et des technologies | 0,001 | 0,003 |
| Communication savante | 0,003 | 0,004 |
| Science ouverte | 0,004 | 0,003 |
| Intégrité de la recherche | 0,004 | 0,005 |
| Charge utile insuffisante (le modèle a refusé de juger) | 0,008 | 0,001 |
Scores machine (provisoires)
Les deux têtes enseignantes du modèle étudiant, lues sur ce travail. Un score ordonne la base pour la relecture; il n'affirme jamais une catégorie, et le statut de validation accompagne chaque rangée tel quel.
Scores de référence d'un modèle non mature (critères de maturité non atteints, 7 itérations). Un score ordonne; il n'affirme jamais une catégorie.
score_only:v0-immature-baseline · tel quel depuis la passe de notation : score_only signifie que le nombre peut ordonner les travaux, et qu'aucune étiquette de catégorie n'en découleClassification
machine, non validéePrédiction automatique; un appel candidat d’une seule source (Gemma direct ou Codex distillé), pas un consensus.
Le détail, modèle par modèle et score par score, se trouve en fin de page sous « Comment cette classification a été obtenue ».