The ROBINS‐I and the NOS had similar reliability but differed in applicability: A random sampling observational studies of systematic reviews/meta‐analysis
Bibliographic record
Abstract
OBJECTIVE: There is a lack of evidence on the usage of the quality assessment tool-the Risk Of Bias In Nonrandomized Studies-of Interventions (ROBINS-I). This article aimed to measure the reliability, criterion validity, and feasibility of the ROBINS-I and the Newcastle-Ottawa Scale (NOS). METHODS: A sample of systematic reviews or meta-analyses of observational studies were selected from Medline (2013-2017) and assessed by two reviewers using ROBINS-I and the NOS. We reported on reliability in terms of the first-order agreement coefficient (AC1) statistic. Correlation coefficient statistic was used to explore the criterion validity of the ROBINS-I. We compared the feasibility of the ROBINS-I and NOS by recording the time to complete an assessment and the instances where assessing was difficult. RESULTS: Five systematic reviews containing 41 cohort studies were finally included. Interobserver agreement on the individual domain of the ROBINS-I as well as the NOS was substantial with a mean AC1 statistic of 0.67 (95% CI: 0.50-0.83) and 0.73 (95% CI: 0.65-0.81), respectively. The criterion validity of the ROBNS-I was moderate (K = 0.52) against NOS. The time in assessing a single study by ROBINS-I varied from 7 hours initially to 3 hours compared with 30 minutes for the NOS. Both reviewers rated "bias due to departure from the intended interventions" the most time-consuming domain in the ROBINS-I, items in the NOS were equal. CONCLUSIONS: The ROBINS-I and the NOS seem to provide the same reliability but vary in applicability. The over-complicated feature of ROBINS-I may limit its usage and a simplified version is needed.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Direct model labels (unvalidated)
Per-model category and study-design labels from the labeling rounds. They are machine output, unvalidated, and the disagreement between models ships as data. No study design here is MEDLINE-validated yet.
| Model arm | Categories | Study design | Confidence |
|---|---|---|---|
| gemma | Metaresearch Domain: Methods · Genre: Empirical About the Canadian research system: no · About a Canadian topic: no | Observational | low |
| gpt | Metaresearch Domain: Methods · Genre: Empirical About the Canadian research system: no · About a Canadian topic: no | Observational | high |
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.352 | 0.525 |
| Meta-epidemiology (narrow) | 0.003 | 0.002 |
| Meta-epidemiology (broad) | 0.013 | 0.037 |
| Bibliometrics | 0.013 | 0.012 |
| Science and technology studies | 0.002 | 0.003 |
| Scholarly communication | 0.005 | 0.004 |
| Open science | 0.003 | 0.005 |
| Research integrity | 0.003 | 0.002 |
| Insufficient payload (model declined to judge) | 0.003 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedLabeled directly by 2 models reading the full record.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".