Whom Do Bureaucrats Believe? A Randomized Controlled Experiment Testing Perceptions of Credibility of Policy Research
Bibliographic record
Abstract
More than ever before, analysts in government have access to policy‐relevant research and advocacy, which they consume and apply in their role in the policy process. Academics have historically occupied a privileged position of authority and legitimacy, but some argue this is changing with the rapid growth of think tanks and research‐based advocacy organizations. This article documents the findings from a randomized controlled survey experiment using policy analysts from the British Columbia provincial government in Canada to systematically test the source effects of policy research in two subject areas: minimum wage and income‐splitting tax policy. Subjects were asked to read research summaries of these topics and then assess the credibility of each article, but for half of the survey respondents the affiliation/authorship of the content was randomly reassigned. The experimental findings lend evidence to the hypothesis that academic research is perceived to be substantially more credible than think tank or advocacy organization research, regardless of its content. That increasingly externalized policy advice systems are not a pluralistic arena of policy research and advice, but instead subject to powerful heuristics that bureaucrats use to sift through policy‐relevant information and advice, demands added nuance to both location and content‐based policy advisory system models.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.054 | 0.136 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.003 | 0.006 |
| Scholarly communication | 0.004 | 0.003 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.004 | 0.004 |
| Insufficient payload (model declined to judge) | 0.012 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".