Comparing the reliability and validity of youth‐reported checklists and standardized interviews for categorical measurement of emotional and behavioral problems
Bibliographic record
Abstract
INTRODUCTION: Self-completed checklists measuring youth mental health problems produce dimensional scale scores and can be converted to categorical classifications representing the presence/absence of psychopathology. We test whether categorical classifications from scale scores are equivalent psychometrically to categorical classifications of the same problems obtained by lay-administered standardized structured diagnostic interviews. METHODS: The sample of n = 325 youth aged 12-18 (44% male) and their parent/caregivers come from combined test-retest reliability studies conducted in Ontario, Canada, from 2011 to 2015. Ontario Child Health Study Emotional Behavioural Scales-Brief Version (OCHS-EBS-B) scores converted to categorical classifications of emotional and behavioral problems were compared with interview classifications. We test hypotheses of statistical equivalence and inferiority, using a confidence interval approach to detect if differences lie within the smallest effect size of interest of ±0.18. We compare categorical classifications on: (1) test-retest reliability (ҡ), (2) content validity (between-instrument agreement), and (3) construct validity (strength of association with three mental health-related constructs). RESULTS: Average test-retest reliabilities were 0.695 (checklists) and 0.670 (interviews). The reliability of checklist emotional problem classifications was not inferior to interview classifications and the difference in reliability between instruments for behavioral problems was small (-0.036). Average between-instrument agreement was ҡ = 0.586 (observed) and ҡ = 0.841 (corrected for attenuation due to measurement error) indicating high content overlap. Statistical equivalence criteria were met in 5 of 6 construct validity comparisons. CONCLUSIONS: Categorical classifications of emotional and behavioral problems from youth-reported checklists are, on balance, equivalent to interview classifications. Checklists represent a simple, brief, inexpensive alternative to interviews.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.022 | 0.050 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".