REPOPA indicators for evidence-informed policy making validated by an international Delphi study
Bibliographic record
Abstract
A total number of 23 measurable REPOPA indicators were organised in four thematic sets - 1) Human resources-Competences & Networking, 2) Documentation-Retrieval/Production, 3) Communication & Participation, 4) Monitoring & Evaluation - and evaluated by means of an international Delphi study. 76 panelists from six European countries (Romania, Italy, Denmark, UK, the Netherlands, Finland) and international organizations, chosen for being researchers or policy makers in public health and across sectors, had to rate relevance and feasibility for each indicator, comment their ratings and propose new indicators by means of two internet-based Delphi rounds. Most indicators were directly validated in the first round, reaching immediately consensus on their high feasibility and relevance; remaining indicators were rated again in the second round, where panelists considered first round scores and comments. Finally 19 out of 23 initial indicators and six out of eight newly suggested indicators were accepted, with a validated list of 25 indicators for EIPM as the final output. Insights emerged as a result of panelists’ ratings and comments, e.g. involvement of researchers was considered essential in all policy phases, while the role of other stakeholders in policy, although considered crucial, raised more discussion, leading to their final exclusion from the policy evaluation phase; allocating budget for EIPM was considered not feasible in most cases, except when devoted to methodologies to engage and consult stakeholders; acquiring evidence from documents was easily welcomed, but citing results from peer reviewed journals and producing evidence on the policy raised more perplexities because of policy makers’ lack of time and familiarity with this kind of literature. Key message: The international Delphi process helped to validate 25 measurable REPOPA indicators aimed at fostering EIPM and produced collective knowledge by means of interaction among policy makers and researchers
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.054 | 0.019 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.002 |
| Open science | 0.002 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; both teacher heads agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".