Consistency and Sensitivity Evaluation of the Saudi Arabia Mental Health Surveillance System (MHSS): Hypothesis Generation and Testing
Bibliographic record
Abstract
Background Public health surveillance systems should be evaluated periodically, and the evaluation should include recommendations for improving the system’s quality and efficiency. Each surveillance system may have a unique situation in which evaluating its quality depends on its methodology, aims, and other factors, such as the frequency of repeating the survey in the case of survey-based surveillance. Objective As the consistency of the surveillance system to capture demographic data and its sensitivity to monitor the intended health-related event are important indicators of the quality of the surveillance system, the aim of this article is to evaluate the Saudi Arabia Mental Health Surveillance System (MHSS) in terms of consistency and sensitivity via the scientific hypothesis testing process. Methods The quality of the MHSS was assessed by examining (1) the consistency of the main demographic variables and (2) the sensitivity to changes in score between the 2 mental health screening tools used in the MHSS and between the 3 waves collected in 3 consecutive months. The assessment uses all data collected via the MHSS between May 2020 and July 2020. The first null hypothesis predicted there were differences between the distributions of the demographic variables between the 3 waves. The second predicted there were no differences between the scores of the Patient Health Questionnaire 9 (PHQ-9) and the Generalized Anxiety Disorder 7-item scale (GAD-7) between the 3 waves. Results In terms of sampling variables (age, gender, and region), there were no significant differences between the 3 waves in age, using one-way ANOVA, nor in gender and region, using the chi-square test. In addition, there were no significant differences between the 3 waves in all other demographic variables, except in the income variable. However, in terms of the PHQ-9 score, the one-way ANOVA (F2,12334=8.05; P<.001) showed significant differences between waves. Similarly, significant differences between waves were found in the GAD-7 score (F2,12334=7.09; P=.001). Conclusions The MHSS showed a consistent distribution of the sample demographic variables, while being sensitive to the changes in mental health scores across waves. The MHSS can generate an acceptable level of consistency and sensitivity to monitor mental health trends. International Registered Report Identifier (IRRID) RR2-10.2196/23748
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.458 | 0.601 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.008 |
| Bibliometrics | 0.005 | 0.004 |
| Science and technology studies | 0.002 | 0.007 |
| Scholarly communication | 0.004 | 0.004 |
| Open science | 0.003 | 0.004 |
| Research integrity | 0.002 | 0.002 |
| Insufficient payload (model declined to judge) | 0.002 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".