PROTOCOL: Do evidence summaries increase health policy‐makers’ use of evidence from systematic reviews? A systematic review protocol

The problem Systematic reviews are becoming increasingly important for policy-makers making decisions about health (Lavis et al., 2006; Petticrew et al., 2004; Welch et al., 2012). The World Health Organization (WHO) defines health as “a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity”(WHO, 2017). Systematic reviews offer many potential benefits to policy-makers, including identifying interventions that are effective or not effective, are considered to have lower risk of bias than other studies, and offer more confidence in results than single studies (Lavis et al., 2006a). However, health policies are often made without the use of research evidence (Oxman et al., 2009). Barriers to the use of research, specifically systematic reviews, in policymaking have been identified (Oliver et al., 2014). Most systematic reviews are written using technical language, are too long, and lack the contextual information important for policy-makers and other users making decisions about how to use the evidence in their decision making (Lavis et al., 2005a; Rosenbaum et al., 2011; Sheldon, 2005). Strategies to promote the use of research evidence to policy-makers are required, and evidence summaries have been suggested as a facilitator to evidence-informed decision making (Bunn & Sworn, 2011). The intervention Systematic review summaries may be called evidence summaries, policy briefs, briefing papers, briefing notes, evidence briefs, abstracts, summary of findings, or plain language summaries (Adam et al., 2014). Within the Cochrane Collaboration, the Evidence Aid Project was developed in response to the 2004 Indian Ocean Tsunami as a means of providing decision makers and health practitioners ‘on the ground’ with summaries of the best available evidence needed to respond to emergencies and natural disasters (Kayabu & Clarke, 2013). In addition to Evidence Aid, there are many organizations that develop and disseminate evidence summaries for different populations or subsets of decision makers. For example, SUPPORT Summaries were developed for policy-makers in low- and middle-income countries making decisions about maternal and child health programs and interventions (www.support-collaboration.org). Health Systems Evidence provides policy briefs for policy-makers making health systems decisions (www.healthsystemsevidence.org/). Communicate to vaccinate (COMMVAC) is creating user friendly summaries to translate evidence on vaccination communication for policy makers and the community in low- and middle-income countries (LMICs) (http://www.commvac.com). Rx for change is a searchable database for evidence about intervention strategies to alter behaviours of health technology prescribing, practice, and use (www.cadth.ca/resources/rx-for-change). Harvesting Evidence summarizes evidence on health systems and/or immunization for decision-making and implementation (http://www.harvesting-evidence.org). In fact, Lavis et al. identified 16 organizations involved in the production of summaries for policy makers in low- and middle-income countries (Adam et al., 2014). How the intervention might work Systematic review summaries consist of summarized evidence from systematic reviews intended to assist policy-makers in understanding the systematic review evidence and using it in their decision making. These interventions may include structured summaries (e.g. SUPPORT summaries, Evidence Aid), policy briefs which are based on systematic reviews (e.g. Health Systems Evidence), and plain language summaries, structured abstracts, and Summary of Findings tables (e.g. Cochrane reviews). These may be provided in print or web-based formats and are aimed at policy-makers, and other decision makers making decisions about health. The summaries may include information about the context in which the studies were conducted, the applicability of the results (e.g. SUPPORT Summaries comment on the relevance of the findings for disadvantaged communities), as well as the findings, methods and conclusions. A needs assessment conducted by Evidence Aid found that systematic review summaries could improve understanding of users (i.e. non-governmental organizations (NGOs), health care providers) so that they can make decisions on the applicability of the findings to their local setting (Kayabu & Clarke, 2013). These user-friendly formats highlight the policy-relevant information and allow policy makers to quickly scan the document for relevance (Lavis et al., 2005a; Lavis et al., 2006a). Evidence suggests that policy makers are more likely to use systematic reviews when the evidence is provided in a timely manner, and aligns with interests, values, and political goals of policymakers (Lavis et al., 2005a; Lavis et al., 2005b). Evidence summaries may increase the use of systematic review evidence by policymakers because they fulfil these by: 1) providing “user friendly” and plain language summaries of the evidence, 2) providing evidence “at-a-glance” with links to the complete systematic reviews, and 3) focusing on policy-relevant topics (Adam et al., 2014; Lavis et al., 2009). In addition, they may improve access to systematic reviews, because most organizations make summaries available freely through online databases and repositories (Adam et al., 2014). Why it is important to do the review Interest in the production and use of systematic review summaries is increasing, as evidenced by the growing number of organizations developing and disseminating them (Adam et al., 2014). However, evidence on the usefulness and effectiveness of systematic review derivatives is lacking. Previously conducted systematic reviews have looked at interventions to increase the use of systematic reviews among decision makers, however, these have focused on the use of complete systematic reviews in decision-making, and none identified focused specifically on derivatives of systematic reviews. For example, Murthy et al. conducted a systematic review examining the effectiveness of interventions for improving the use of systematic reviews in decision-making by health system managers, policy-makers, and clinicians (Murthy et al., 2012). Eight studies were included and the authors concluded that information provided as a single, clear message may improve evidence-based practice, but increasing awareness and knowledge of systematic review evidence might require a multi-faceted intervention. Similarly, Perrier et al. conducted a systematic review of interventions encouraging the use of systematic reviews by health policymakers and managers (Perrier et al., 2011). Four studies were included in the systematic review and the authors concluded that future research should identify how systematic reviews are accessed and the formats used to present the information. Finally, a review by Wallace et al. found that the facilitators to increase systematic review use by policymakers included description of benefits as well as harms and costs, and using a 1:3:25 staged approach to evidence summaries (Wallace, Byrne, & Clarke, 2012). However, none of these reviews were focused on summaries created from systematic reviews. In addition, we will focus on studies of evidence summaries for health policy-makers and health system managers making decisions on behalf of a large jurisdiction or organization but will not include studies related to decision making for an individual person or patient. The contribution of this review To the best of our knowledge, this is the first systematic review to assess the use of systematic review summaries in policy-making. The results of this review will inform researchers and systematic review summary developers of the best way to present the evidence to ensure that evidence summaries fulfill their goal of informing policy-makers with the best possible evidence needed to make health-related decisions. The objectives of this review are to 1) assess the effectiveness of evidence summaries on health policy-makers’ use of the evidence and 2) identify the most effective components of the summaries for increasing policy-makers’ use of the evidence. Characteristics of studies relevant to the objectives of the review Criteria for inclusion and exclusion of studies in the review Interventions We will include studies examining the effects of any type of “friendly front end”, “evidence summary”, or “policy brief” or other product derived from systematic reviews or guidelines based on systematic reviews that presents evidence in a summarized form to health policy-makers and health system managers. We will exclude studies in which evidence summaries are one component of a multi-component intervention. For this review, we define a systematic review as meeting the following criteria: clear inclusion and exclusion criteria, a comprehensive and systematic search strategy, explicit and reproducible methodology, an assessment of the validity of the findings of the included studies, and a systematic presentation and synthesis or meta-analysis (if possible) of the findings of the included studies (Campbell Collaboration, n.d.; Green et al., 2011). Interventions must include a summary of a systematic review and be actively “pushed” to target users, meaning that the summaries are packaged for decision makers and made accessible (Lavis et al., 2006a). For example, a potentially included study used an intervention that evaluated the effectiveness of friendly front ends by assessing changes in policy-maker beliefs (Beynon et al., 2012). An example of a study that would be excluded assessed the views of policymakers on how systematic reviews can be promoted within a low- and middle-income country (Yousefi-Nooraie et al., 2009). We will include any comparisons including active comparators (e.g. other summary formats) or no intervention. This will allow us to assess both whether summaries increase health policy-makers’ use of the evidence when compared to no summaries or different types of summaries as well as whether some formats are more effective. We will assess studies using an active comparator separately from those with no intervention in the control group. Participants We will include studies which include health policy-makers at all levels (including: civil society organization staff, non-governmental organization staff, local government staff, regional government staff, federal government staff) and health system managers making decisions on behalf of a large jurisdiction or organization (Perrier et al., 2011). We will not include studies related to decision making for an individual person or patient. For the purposes of this review, we define ‘health policy-makers’ as those responsible for making decisions about healthcare policies and programs which are those intended to restore or maintain physical, mental, or emotional well being (WHO, 2017). Outcomes Primary outcomes Secondary outcomes We recognize that some studies may use different terms to describe these outcomes. For example, the term ‘satisfaction’ maybe used as an umbrella term to capture relevance, usability, and desirability. These outcomes will be assessed by the team and categorized according to the above list. Research methods/designs We will include randomised controlled trials (RCTs), non-randomised controlled trials (NRCTs), controlled before-after CBA (studies), and interrupted time series (ITS) studies. Search strategy for finding eligible studies An Information Specialist will help develop the search strategy using the PRESS Guideline (Sampson et al., 2008). We will build on the search strategy used by Perrier et al. and Murthy et al. in their systematic reviews of interventions to encourage the use of systematic reviews by health managers and policy-makers.(Murthy et al., 2012; Perrier et al., 2011) The search conducted by Perrier et al. identified 11,297 records (after removing duplicates) and included four papers reporting two studies. This search included the following databases: Medline, EMBASE, CINAHL, Cochrane Central Register of Controlled Trials, Cochrane Methodology Register, Health Technology Assessment Database, and LISA. We will modify these search strategies and expand this search by including additional databases, as suggested by John Eyres, of the International Initiative for Impact Evaluation (3ie) and the Campbell International Development Review Group. These include databases such as Global Health (CABI), Global Health Library (from WHO), Popline, Africa-wide, Public Affairs Information Service, Worldwide Political Science Abstracts, Web of Science, DfiD (Research for Development Database) (See Appendix 1 for MEDLINE search strategy). We will search websites of research groups and organizations producing evidence summaries to identify unpublished studies evaluating the effectiveness of the systematic review derivatives in increasing policy-makers’ understanding (e.g. Health Systems Evidence, the Canadian Agencies for Health Technology Assessment, SUPPORT Summaries). We will check reference lists of relevant studies to identify additional studies. We will contact researchers to identify ongoing and completed/published work. We will report the results of the search using the PRISMA flow diagram. Data extraction and study coding procedures Two reviewers will independently screen titles and abstracts to identify relevant studies meeting the pre-specified inclusion criteria. The full text of potentially included studies will be screened independently by two authors. Data extraction and quality assessment will be conducted independently and in duplicate. We will use Covidence software (https://www.covidence.org/) for screening of studies. All completed studies will be included if they meet the inclusion criteria listed above. The data extraction form will be pre-tested, and will include factors related to the population, intervention, comparison, and outcomes. The data will be extracted independently in duplicate by two reviewers using a structured Excel sheet and will be piloted on ten articles. Disagreements on extractions will be resolved by discussion and with a third member of the research team when necessary. Risk of bias The methodological quality will be specifically examined using the risk of bias tools from the Cochrane Handbook for randomised trials and the Effective Practice and Organization of Care (EPOC) Review Group criteria for interrupted time series and controlled before after studies (Ballini et al., 2011; Higgins & Green, 2011) and the Risk of Bias in Non-randomised Studies – of Interventions (ROBINS-I) tool (Sterne et al., 2014) . Risk of bias will be assessed independently, in duplicate, by two authors and any discrepancies will be resolved by consensus and consultation with a third author, when necessary. Synthesis procedures and statistical analysis Effect estimates and confidence intervals for individual studies will be calculated (where possible) irrespective of whether a pooled effect estimate is calculated. When it is not possible to combine the data, we will present the results for each study separately. When possible, any studies with cluster allocation (e.g. cluster-randomised trials, cluster-allocated controlled before and after studies, and interrupted time series) analyses with errors in the unit of analysis will be adjusted using the variance inflation factor, as described in the Cochrane Handbook, if the necessary data can be obtained from the study authors. We will obtain ICC from other similar studies with similar outcomes if the ICC is not published (eg by checking the Aberdeen website of ICCs, http://www.abdn.ac.uk/hsru/research/delivery/behaviour/methodological-research/, or the Campbell Collaboration website of ICCs for education). Sensitivity analyses will be used to assess the effects of incorporating these corrected analyses in our analysis. We will attempt to contact the contact author of the studies by email for any missing data. If more than 10 studies are included we will use funnel plots to explore publication bias. Where appropriate, results will be synthesised using meta-analysis following the guidance outlined in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins & Green, 2011). When it is possible to combine studies, dichotomous outcomes will be reported as relative risks. Continuous outcomes will be reported as weighted mean differences. If an outcome has been reported in different scales (e.g. understanding), and we consider the scales to measure a similar construct, standardised mean differences will be used to summarise the data. We will analyse studies with an active comparator separately from those providing no intervention to the control group. Non-randomised studies will be meta-analyzed separately from RCTs. When results cannot be pooled, we will present a narrative summary of the results. We will analyze the results of qualitative data from the included quantitative studies, when possible, to understand the perceptions and attitudes regarding the components of the summaries that were considered the most useful. We will use the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach to assess the quality of evidence for the outcomes reported in this review (Guyatt et al., 2011). Subgroup analysis and investigation of heterogeneity Evidence summaries, like systematic reviews, that seek to inform decisions in a neutral way should not contain recommendations. Therefore, we will conduct a subgroup analysis to compare the effectiveness of summaries that provide recommendations and those without them. Sensitivity analysis The effect of including studies assessed as high risk of bias or studies in which there were unit of analysis errors that could not be re-analyzed will be considered in sensitivity analysis. This work is funded by JP's CIHR Doctoral Research Award and a Campbell Systematic Review Award. The funders had no role in the development of this protocol. The authors have no competing interests to declare. Lead review author: The lead author is the person who develops and co-ordinates the review team, discusses and assigns roles for individual members of the review team, liaises with the editorial base and takes responsibility for the on-going updates of the review. Training and pilot testing on the inclusion criteria: Completion by May 20 Searches for eligible studies: Completion by May 25 Screening the results from the literature search: Completion by June 21 Training and pilot testing the study coding procedure: Completion by June 30 Extraction of data from eligible research reports: Completion by July 15 Statistical analysis: completion by July 31, 2016 Preparation of the final review report: completion by August 15, 2016 The review will be updated every two years. JP will be responsible for leading the updates. By completing this form, you accept responsibility for preparing, maintaining and updating the review in accordance with Campbell Collaboration policy. The Campbell Collaboration will provide as much support as possible to assist with the preparation of the review. A draft review must be submitted to the relevant Coordinating Group within two years of protocol publication. If drafts are not submitted before the agreed deadlines, or if we are unable to contact you for an extended period, the relevant Coordinating Group has the right to de-register the title or transfer the title to alternative authors. The Coordinating Group also has the right to de-register or transfer the title if it does not meet the standards of the Coordinating Group and/or the Campbell Collaboration. You accept responsibility for maintaining the review in light of new evidence, comments and criticisms, and other developments, and updating the review at least once every five years, or, if requested, transferring responsibility for maintaining the review to others as agreed with the Coordinating Group. The support of the Campbell Collaboration and the relevant Coordinating Group in preparing your review is conditional upon your agreement to publish the protocol, finished review and subsequent updates in the Campbell Library. Concurrent publication in other journals is encouraged. However, a Campbell systematic review should be published either before, or at the same time as, its publication in other journals. Authors should not publish Campbell reviews in journals before they are ready for publication in the Campbell Library. Authors should remember to include a statement mentioning the published Campbell review in any non-Campbell publications of the review. I understand the commitment required to undertake a Campbell review, and agree to publish in the Campbell Library. Signed on behalf of the authors: Form completed by: Jennifer Petkovic Date: 1 August 2017 Appendix 1: MEDLINE Search Strategy 1 ((systematic review$ or methodolog$ review$ or quantitativ$ review$ or qualitativ$ review$ or overview$ or synthes$ or metasynthes$ or megasynthes$) adj5 (decisionmak$ or decision-mak$ or policy-mak$ or policymak$ or policy decision$ or health$ polic$ or health$ manag$ or action$ or commission* or purchas* or procur* or budget hold* or budgethold* or service provi*or practice or application or implement$ or utili?ation or utili?ing or utili$ or disseminat$ or summar$ or hospital* decision* or treatment plan* or patient care or patient care or healthcare or health care or clinical decision* or pathway* or algorithm*)).ti,ab. (30531) 2 (((systematic adj2 (review* or overview* or synthesis or literature review* or evidence review*)) or methodolog* review* or quantitativ* review* or qualitative review* or overview or synthes* or metasynthes* or megasynthes*) adj5 (policy or policies or decision*)).ti. (400) 3 ((gap or gaps) adj7 ((knowledge or research or evidence or trial or result) adj2 practice)).ti,ab. (972) 4 1 or 2 or 3 (31697) 5 randomized controlled trial.pt. (409861) 6 controlled clinical trial.pt. (90286) 7 randomized.ab. (339312) 8 placebo.ab. (167495) 9 clinical trials as topic/ (175364) 10 randomly.ab. (244542) 11 trial.ti. (147363) 12 intervention*.ti. (96258) 13 or/5-12 (1074398) 14 State Medicine/ (52025) 15 exp Purchasing, Hospital/ (5754) 16 Contracts/ (2870) 17 exp Contract Services/ (12176) 18 exp Organizational Innovation/ (23682) 19 Insurance, Health/ or exp Managed Care Programs/ or Medicare/ (96098) 20 (commissioning or commissioner$).ti,ab. (4202) 21 (purchasing or purchaser$).ti,ab. (8743) 22 (procurement or procurer$).ti,ab. (6959) 23 (budget-holder$ or budgetholder$).ti,ab. (48) 24 (service adj2 (development or developer$ or provision or provider$)).ti,ab. (12067) 25 ((investment or budget or purchas$ or service) adj3 priorit$).ti,ab. (554) 26 priorit$ setting.ti,ab. (1563) adj3 or or 30 exp exp Health exp exp of or knowledge or or adj5 or disseminat$ or implement$ or or or application or or or synthes$ or or or or or or or or or or or or or or or or findings or or adj5 or disseminat$ or implement$ or or or application or or or synthes$ or or or or or or or or or or or or or evidence adj5 or or policy or policies or practice or or knowledge or or adj5 or or or or or or or or or or or or or or research or evidence or knowledge or adj5 or or policy-mak$ or policymak$ or health$ manag$ or health$ polic$ or action$ or practice or policy or using or or or or adj5 or Information adj5 or of Health or knowledge or or evidence or information or adj5 or summar$ or or overview$ or or synthes$ or or or or or or or or or or or or or or or or or or or or or or and of protocol disease 4 and 13 and or or or or or community or or or or or or or or or or or or or or or or or or multi-component or or or or or or or or or or or or or or or or care or or or or or or or or or or or or or or or or or or adj5 or or adj3 or (after adj3 trial.ti. or adj3 or (after or or or or or or or or adj3 or study or trial or adj2 adj3 or or or four or five or or or or or ten or or or or or or trial or controlled clinical trial or or or or or adj3 or or or or or or or or not clinical trial or randomized controlled studies as topic/ or or or or adj5 or adj2 or or or not or randomized controlled trial.pt. or or or or or or or or or or exp not not 4 and and or or

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.