Non‐randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions
Bibliographic record
Abstract
The terms applicability, generalizability, external validity and transferability are related, sometimes used interchangeably and have in common that they lack a clear and consistent definition in the classic epidemiological literature. However, all of these terms generally describe one overarching theme: whether or not available research evidence can be directly utilized to answer the healthcare questions at hand, ideally supported by a judgment about the degree of confidence for this utilization. This concept has been called directness. The objectives of this paper were to delineate how non-randomized studies (NRS) inform judgments in relation to directness and the concepts that it encompasses in the context of systematic reviews. We will briefly review what is known and describe the theoretical and practical issues as well as offer guidance to those tackling the challenges of judging directness and using research evidence to answer healthcare questions with evidence from NRS. In particular, we suggest a framework in which authors can use NRS as a complement, sequence or replacement for randomized controlled trials (RCTs) by focusing on judgments about the population, intervention, comparison and outcomes. Authors of systematic reviews will use NRS to complement judgments about the inconsistencies, the rationale and credibility of subgroup analysis, the baseline risk estimates for the determination of absolute benefits and downsides, and the directness of surrogate outcomes. This evidence includes contextual or supplementary evidence. Authors of systematic review and other summaries of the evidence use NRS as sequential evidence to provide evidence when insufficient evidence is available for an outcome from RCTs, but NRS evidence is available (e.g., long-term harms). Use of evidence from NRS may also serve to replace RCT evidence when NRS provide equivalent (or potentially higher) confidence in the evidence (i.e. quality) compared to indirect evidence from RCTs. These judgments will be made in the context of other domains that influence the overall quality of the body of evidence, including the risk of bias, publication bias (i.e. limitations in the detailed study design and execution), inconsistency, imprecision and factors that increase our confidence in effects. This article will support systematic reviewers in their interaction with decision makers, that is, those who use the systematic review to develop guidelines, address health policy makers, and make clinical decisions, by making these judgments transparent. Copyright © 2013 John Wiley & Sons, Ltd.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.622 | 0.849 |
| Meta-epidemiology (narrow) | 0.006 | 0.008 |
| Meta-epidemiology (broad) | 0.020 | 0.021 |
| Bibliometrics | 0.044 | 0.038 |
| Science and technology studies | 0.005 | 0.026 |
| Scholarly communication | 0.025 | 0.031 |
| Open science | 0.012 | 0.018 |
| Research integrity | 0.020 | 0.021 |
| Insufficient payload (model declined to judge) | 0.013 | 0.004 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".