REPOPA indicators for evidence-informed policy making validated by an international Delphi study
Bibliographic record
Abstract
A total number of 23 measurable REPOPA indicators were organised in four thematic sets - 1) Human resources-Competences & Networking, 2) Documentation-Retrieval/Production, 3) Communication & Participation, 4) Monitoring & Evaluation - and evaluated by means of an international Delphi study. 76 panelists from six European countries (Romania, Italy, Denmark, UK, the Netherlands, Finland) and international organizations, chosen for being researchers or policy makers in public health and across sectors, had to rate relevance and feasibility for each indicator, comment their ratings and propose new indicators by means of two internet-based Delphi rounds. Most indicators were directly validated in the first round, reaching immediately consensus on their high feasibility and relevance; remaining indicators were rated again in the second round, where panelists considered first round scores and comments. Finally 19 out of 23 initial indicators and six out of eight newly suggested indicators were accepted, with a validated list of 25 indicators for EIPM as the final output. Insights emerged as a result of panelists’ ratings and comments, e.g. involvement of researchers was considered essential in all policy phases, while the role of other stakeholders in policy, although considered crucial, raised more discussion, leading to their final exclusion from the policy evaluation phase; allocating budget for EIPM was considered not feasible in most cases, except when devoted to methodologies to engage and consult stakeholders; acquiring evidence from documents was easily welcomed, but citing results from peer reviewed journals and producing evidence on the policy raised more perplexities because of policy makers’ lack of time and familiarity with this kind of literature. Key message: The international Delphi process helped to validate 25 measurable REPOPA indicators aimed at fostering EIPM and produced collective knowledge by means of interaction among policy makers and researchers
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.239 | 0.228 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.003 |
| Bibliometrics | 0.013 | 0.009 |
| Science and technology studies | 0.003 | 0.004 |
| Scholarly communication | 0.007 | 0.006 |
| Open science | 0.003 | 0.014 |
| Research integrity | 0.002 | 0.003 |
| Insufficient payload (model declined to judge) | 0.011 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".