How Research Influences Policy Makers: Still Hazy After All These Years

Libraries have been written about the theory and practice of public policy making. Yet, this enormous scholarship has proved insufficient to lift the veil of mystery and idiosyncrasy that shrouds the art of decision making. The heady ambition to turn both clinical practice and health policy into evidence-based bastions of rationalist decision making has been downgraded; the vocabulary is now “evidence-informed,” and the realm of admissible evidence has been greatly expanded to include preferences, political contingencies, and psychology ( 1 ). This newfound conceptual modesty and nuance does not suggest that we should abandon efforts to understand decision-making processes and to enhance the role of research-based evidence in policy. It merely confirms the complexity, contingency, and messiness of the terrain. In this issue of the Journal, Brownson et al. ( 2 ) have boldly entered this murky territory. They attempt to identify the potential impact of carefully crafted policy briefs on three groups of policy makers: legislators, legislative committee staffers, and public health executives. They reviewed a wide literature and created four policy briefs—data- or story-focused briefs that included state- or local-level data—on the topic of mammography screening. They persisted with recruitment efforts to engage hundreds of participants in six states, and given the topic and the terms of engagement, the 35% response rate is impressive. Their analytic objective was to identify which approaches are most effective in which circumstances and which characteristics of both the policy briefs and their recipients predict influence and uptake. It is therefore a joyless duty to report that for all its creativity and effort, the study is less of an advance in our understanding of how evidence, mode of presentation, and policy making interact than one would have hoped. Its ambitions were not fully realized because of some critical design limitations that raise serious doubts about what the findings tell us, and whether they rest on a solid foundation. The first serious design issue is the choice of subject matter. If one wants to perform an independent test of policy brief types among related but different audiences, it would be wise to choose a topic that respondents are unlikely to be informed about or invested in. It is unimaginable that a substantial proportion of respondents were unaware of mammography screening and/or neutral as to its utility. If the briefs had been designed to present the different perspectives on mammography, the study could have revealed how preexisting views and conclusions affect the reception of potentially countervailing information. Instead, all of the briefs promote the same entirely pro-mammography perspective, but nothing about the hazards of interventions occasioned by ambiguous screening results or any acknowledgment that while there may be policy consensus that screening women older than 50 years is a sound strategy, scientific controversy persists. One can counter that it is of no consequence whether the substantive content of the policy briefs is contestable—the study was designed to elicit views on clarity and use, not on scientific accuracy or comprehensiveness—but the very prominence of mammography in public discourse is an avoidable contaminant in a laboratory already fraught with uncertainty. This leads to a second issue: the definition of “use.” The questionnaires simply asked whether the respondents intended to use the brief. We do not know what “use” means. Does it mean that the brief created or altered either their understanding of the issue or their inclination to affect or create policy? Clear and persuasive information can confirm or change opinion about what to do and the urgency of doing it. Understanding how the policy brief “works” requires some understanding of respondents’ baseline views and how the new or repackaged information affects them. It also requires knowledge of other sources of information and advocacy on the mammography issue. For some, the policy brief may be the first and only thing they have ever read on the subject; for others, it may be the first effective compilation of evidence in a single accessible format; and for others, it may be entirely redundant. Yet, the already well informed may report intention to use the briefs because they are already motivated to pursue policy in the area and are happy to have another platoon in the supporting army. Some of the findings are counterintuitive and, as the authors note, inconsistent with previous research. Sometimes novel findings emerge from the ingenuity of the research and confirm why it is important to revisit supposedly settled issues. Sometimes, however, the new findings simply arouse suspicions that something is amiss. It is hard to believe that staffers are looking for stories and that legislators are looking for data. One wonders whether the staff championed the value of narrative in defiance of being typecast as politically naive and cloistered in data. Similarly, legislators may have been wary of appearing to be data-illiterate hostages to anecdote and interest brokering, indifferent to sound science, and overly swayed by the passions. To be clear, this is not to suggest that the respondents were deliberately untruthful; if doctors can genuinely believe that they are following clinical practice guidelines when they in fact are not ( 3 ), it is entirely plausible that policy makers are prone to the same unconscious or partly conscious misunderstandings of their own behaviors. Finally, one wonders if the pathway to deeper understanding of the interaction between research-based evidence and public policy making is paved with qualitative rather than quantitative inquiry. Brownson et al. aspired to breadth and achieved it admirably under the circumstances, but they may have sacrificed critical depth. To get an accurate understanding of how a policy brief might influence policy actors, one would want to know their preexisting knowledge and beliefs about the importance of the issue; their assessment of the policymaking environment; whether they had formed a provisional or strong opinion on policy direction; the extent to which the brief changed their understanding and in which direction; and have a nuanced understanding of what they mean by “use.” It would take a very long and complicated questionnaire to begin to get at these issues in any depth; moreover, one would want to look respondents in the eye when posing questions susceptible to producing a socially desirable response. The study by Brownson et al. is analogous to a clinical trial with ambiguous findings. Those interested in pursuing this important topic should carefully consider how to avoid the problems encountered in the current study, and perhaps go back to the methodological drawing board. Researchers need to be supremely sensitive to the possibility that the data they obtain might lend itself to multiple and even conflicting interpretations and be vigilant about the clarity of questions posed. The relationship between knowledge translation and policy making yields its mysteries grudgingly. It will never be possible to eliminate the chaos inherent in studies of real-world decision making, but we should redouble efforts to create order where we can.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.