Statistical models for multilevel data with “Don’t know” category: implication for program evaluation
Bibliographic record
Abstract
Background: “Don’t know (DK)” category has been increasingly used in surveys of longitudinal research. This creates unique challenges in data analysis and program evaluation. Strategies applying missing data methods may lead to biased and inaccurate estimations and lose valuable information. Objectives: (i) To illustrate advantages of the proposed two-part mixed effects model over other methods for longitudinal outcomes with DKs through simulation; (ii) to apply the proposed model to a mental health program (Project 11) to evaluate the program effects on participants’ awareness and level of connectedness. Methods: We applied a two-part mixed effects model for longitudinal outcome containing DKs. A simulation study was designed to illustrate the advantages of our proposed model over other methods, where different conditions including sample size, DK proportion, correlation strength between DKs and non-DK responses were considered under different DK mechanisms. We also compared the proposed model with other approaches by applying them to a mental health program (Project 11). In the data analysis of Project 11, we further extended the two-part model to account for within-cluster correlations among students within schools and to explore gender differences in program effects. Results: The proposed two-part mixed effects model outperformed other methods (i.e., CCA, SI, and ML) in estimating both intervention and random effects under all DK mechanisms. In contrast, methods disregarding DKs as missing experienced issues in at least some scenarios. Application of the two-part model to Project 11 data suggested significant intervention effects on improving the connectedness among boys (β ̂_11: -0.071, p = 0.049), whereas no significant improvements were observed among girls. Significant correlations were also found between the likelihood of DKs and connectedness level at both student level and school level. Conclusions: The proposed two-part mixed effects model is highly recommended for analyzing data with DK responses, based on the results of both simulations and empirical data analysis. Missing data techniques should be avoided due to potentially biased and/or imprecise estimates and the loss of information conveyed by DK responses.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.360 | 0.645 |
| Meta-epidemiology (narrow) | 0.002 | 0.002 |
| Meta-epidemiology (broad) | 0.006 | 0.010 |
| Bibliometrics | 0.004 | 0.007 |
| Science and technology studies | 0.002 | 0.007 |
| Scholarly communication | 0.006 | 0.010 |
| Open science | 0.010 | 0.006 |
| Research integrity | 0.005 | 0.012 |
| Insufficient payload (model declined to judge) | 0.014 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".