Process Evaluation of a Rapid Evidence Support System Assessment of Ireland’s Department of Health – A Protocol
Bibliographic record
Abstract
Background The Rapid Evidence Support System Assessment (RESSA) was developed by the Global Evidence Commission to evaluate evidence support systems that inform policy decisions. These systems are designed to contextualize existing evidence, guide decision-making, and generate new insights to inform action. As evidence-informed policymaking gains traction globally, it is essential to evaluate these systems’ effectiveness. In Ireland, the Health Research Board, the Department of Health, Evidence Synthesis Ireland, Cochrane Ireland, and the Global Evidence Commission are collaborating to conduct a RESSA within the Department of Health. This process evaluation aims to assess the fidelity, acceptability, and experiences of stakeholders involved in the RESSA, providing insights for refining the methodology. Methods The process evaluation will employ a mixed methods approach, integrating both qualitative and quantitative data collection. It will evaluate the conduct of a RESSA within the Department of Health. Fidelity assessment will examine adherence to the RESSA protocol, while acceptability will be evaluated using the Theoretical Framework of Acceptability, focusing on key stakeholders' attitudes. An exploration of the experiences of participants, capturing both facilitators and barriers to the RESSA’s success will also be conducted. Data analysis will involve thematic analysis and descriptive statistics, aiming to highlight the RESSA’s methodological strengths and areas for improvement. Conclusions This evaluation is expected to provide critical insights into the strengths and limitations of the RESSA methodology, with implications for evidence-informed policymaking. Findings will offer recommendations to enhance the robustness and applicability of the RESSA in Ireland and beyond. Dissemination will include academic publications and reports, contributing to the broader understanding of effective evidence support systems. This process evaluation aims to inform future RESSAs and strengthen the evidence support framework, ensuring better-informed policy decisions at local, national, and international levels.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.468 | 0.547 |
| Meta-epidemiology (narrow) | 0.002 | 0.003 |
| Meta-epidemiology (broad) | 0.003 | 0.006 |
| Bibliometrics | 0.007 | 0.006 |
| Science and technology studies | 0.007 | 0.005 |
| Scholarly communication | 0.012 | 0.008 |
| Open science | 0.006 | 0.009 |
| Research integrity | 0.007 | 0.008 |
| Insufficient payload (model declined to judge) | 0.048 | 0.012 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".