Using a consistency check during data collection to identify invalid responding in an online cannabis screening survey
Bibliographic record
Abstract
BACKGROUND: Inconsistent responding is a type of invalid responding, which occurs on self-report surveys and threatens the reliability and validity of study results. This secondary analysis evaluated the utility of identifying inconsistent responses as a real-time, direct method to improve quality during data collection for an Internet-based RCT. METHODS: The cannabis subscale of the Alcohol, Smoking and Substance Involvement Screening Test (ASSIST) was administered as part of eligibility screening for the RCT. Following the consent procedure, the cannabis subscale was repeated during the baseline interview. Responses were automatically compared and individuals with inconsistent responses were screened out. RESULTS: Nearly half of those initially eligible for the RCT were subsequently screened out for data quality issues (n = 626, 45.3%). Between-group bivariate analysis found that those screened out (OUT) were significantly older (OUT = 39.5 years (SD = 13.9), IN = 35.7 years (SD = 12.9), p < .001), more had annual incomes less than $20,000CND (OUT = 58.3%, IN = 53.0%, p = .047), used cannabis less often in the past 30 days (OUT = 23.3 days (SD = 9.7), IN = 24.8 days (SD = 11.3), p < .006), and had lower total ASSIST scores at screener (OUT = 19.3 (SD = 8.0), IN = 23.8 (SD = 10.4), p < .001) and baseline (OUT = 17.5 (SD = 7.9), IN = 23.3 (SD = 10.3), p < .001) compared to participants who were screened in to the RCT. CONCLUSION: Inconsistent responding may occur at high rates in Internet research and direct methods to identify invalid responses are needed. Comparing responses for consistency can be programmed in Internet surveys to automatically screen participants during recruitment and reduce the need for post-hoc data cleaning.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.312 | 0.371 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.003 |
| Bibliometrics | 0.006 | 0.006 |
| Science and technology studies | 0.003 | 0.002 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.003 | 0.004 |
| Research integrity | 0.002 | 0.003 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".