Assessing the Risk of Bias for Epidemiological Studies Evaluating the Association Between Long-Term Exposures to PM2.5 & All-Cause Mortality
Bibliographic record
Abstract
BACKGROUND AND AIM: Risk of bias (RoB) tools are used in systematic reviews of environmental epidemiology to increase the quality of evidence (Eick et al., 2020). This pilot study aimed to determine the comparability and consistency of specific RoB tools. METHOD: We focused on the World Health Organization (WHO) commissioned systematic review of long-term exposure to PM2.5 and all-cause mortality (Chen et al., 2020). We selected the first 15 references based on the last name of the first author from the alphabetically arranged reference list of included studies. We applied the WHO RoB tool to the selected studies, documenting the rationale for our ratings. We further selected 5 high and 5 low WHO-rated studies, based on the confounding and exposure assessment domains to apply the WHO RoB tool for evaluating the reproducibility of ratings. Finally, we applied the Newcastle Ottawa Scale (NOS) and The Office of Health Assessment and Translation (OHAT) RoB tools to the initially selected 15 studies. RESULTS: The study demonstrated rating variability due to tool differences and assessor interpretation. For example, for the initial selection of fifteen studies, our study ratings for confounding and exposure assessment domains based on the WHO tool contrasted with the WHO analysis but were largely consistent for other domains. Our ROB ratings were largely in agreement with WHO for studies that were rated as high risk of bias for exposure assessment and confounding. However, studies rated by WHO as low risk of bias frequently differed from our assessment for the exposure assessment domain. Lastly, RoB assessments using NOS and OHAT were in agreement with our RoB ratings but inconsistent with the WHO ratings. CONCLUSIONS: While a direct comparison of the RoB tools was challenging, the results suggest that differences in the interpretation of evaluation criteria across tools contribute to variability in ratings which may have significant policy implications.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.017 | 0.025 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".