Whom Do Bureaucrats Believe? A Randomized Controlled Experiment Testing Perceptions of Credibility of Policy Research
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
More than ever before, analysts in government have access to policy‐relevant research and advocacy, which they consume and apply in their role in the policy process. Academics have historically occupied a privileged position of authority and legitimacy, but some argue this is changing with the rapid growth of think tanks and research‐based advocacy organizations. This article documents the findings from a randomized controlled survey experiment using policy analysts from the British Columbia provincial government in Canada to systematically test the source effects of policy research in two subject areas: minimum wage and income‐splitting tax policy. Subjects were asked to read research summaries of these topics and then assess the credibility of each article, but for half of the survey respondents the affiliation/authorship of the content was randomly reassigned. The experimental findings lend evidence to the hypothesis that academic research is perceived to be substantially more credible than think tank or advocacy organization research, regardless of its content. That increasingly externalized policy advice systems are not a pluralistic arena of policy research and advice, but instead subject to powerful heuristics that bureaucrats use to sift through policy‐relevant information and advice, demands added nuance to both location and content‐based policy advisory system models.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.015 | 0.064 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.002 | 0.005 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it