Fit for purpose: perspectives on rapid reviews from end-user interviews
Bibliographic record
Abstract
BACKGROUND: There is increasing demand for rapid reviews and timely evidence synthesis. The goal of this project was to understand end-user perspectives on the utility and limitations of rapid products including evidence inventories, rapid responses, and rapid reviews. METHODS: Interviews were conducted with key informants representing: guideline developers (n = 3), health care providers/health system organizations (n = 3), research funders (n = 1), and payers/health insurers (n = 1). We elicited perspectives on important characteristics of systematic reviews, acceptable methods to streamline reviews, and uses of rapid products. We analyzed content of the interview transcripts and identified themes and subthemes. RESULTS: Key informants identified the following as critical features of evidence reviews: (1) originating from a reliable source (i.e., conducted by experienced reviewers from an established research organization), (2) addressing clinically relevant questions, and (3) trusted relationship between the user and producer. Key informants expressed strong preference for the following review methods and characteristics: use of evidence tables, quality rating of studies, assessments of total evidence quality/strength, and use of summary tables for results and conclusions. Most acceptable trade-offs to increase efficiencies were limiting the literature search (e.g., limiting search dates or language) and performing single screening of citations and full texts for relevance. Key informants perceived rapid products (particularly evidence inventories and rapid responses) as useful interim products to inform downstream investigation (e.g., whether to proceed with a full review or guideline, direction for future research). Most key informants indicated that evidence analysis/synthesis and quality/strength of evidence assessments were important for decision-making. They reported that rapid reviews in particular were useful for guideline development on narrow topics, policy decisions when a quick turn-around is needed, decision-making for practicing clinicians in nuanced clinical settings, and decisions about coverage by payers/health insurers. Rapid reviews may be more relevant within specific clinical settings or health systems; whereas, broad/national guidelines often need a traditional systematic review. CONCLUSIONS: Key informants interviewed in our study indicated that evidence inventories, rapid responses, and rapid reviews have utility in specific decisions and contexts. They indicated that the credibility of the review producer, relevance of key questions, and close working relationship between the end-user and producer are critical for any rapid product. Our findings are limited by the sample size which may have been too small to reach saturation for the themes described.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.372 | 0.517 |
| Meta-epidemiology (narrow) | 0.002 | 0.003 |
| Meta-epidemiology (broad) | 0.002 | 0.003 |
| Bibliometrics | 0.005 | 0.004 |
| Science and technology studies | 0.016 | 0.030 |
| Scholarly communication | 0.030 | 0.033 |
| Open science | 0.006 | 0.032 |
| Research integrity | 0.017 | 0.022 |
| Insufficient payload (model declined to judge) | 0.007 | 0.003 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".