Systematic evidence synthesis as part of a larger process: a response to comments on Berger-Tal et al.

We are encouraged that the prospect of generating systematic reviews and maps (Berger-Tal et al. 2019) has stirred enthusiasm among our peers. The resulting discussion brings up a number of valid points that share a vision for a field with greater internal rigor and external impact. The quality of a review relies heavily on the quality, topical diversity, and open-access availability of the primary literature. We agree with Nakagawa and Lagisz (2019) that promoting reporting standards for experimental studies should be a priority of journals, scientists, and educators. Not only will better reporting help improve the basic level of science in the field, but it will also increase the likelihood that studies can be used as evidence in other contexts (see the EQUATOR Network, www.equator-network.org, for a good example from the field of health research). Assessing the value and the rigor of the science involved in any synthesis is extremely important. We agree with Stewart and Ward (2019) that the potential detrimental weight of a small group of “experts” in the evidence base needs to be considered when investigating potential sources of bias. The ability of scientists to analyze the quality of the science itself is key in making these decisions, and new tools are emerging to aid this process (Nakagawa et al. 2019). Review authors should not be immune from the scrutiny of bias themselves, especially if developing prescriptive evaluations (Stewart and Ward 2019), which is why peer review and a standard reporting format is vital for reducing these biases. Our focus on systematic reviews that adhere to the strict Collaboration for Environmental Evidence (CEE) guidelines does not eliminate the value of other types of literature syntheses. We agree with Nakagawa and Lagisz (2019) that reviews can take various forms that follow the same principles of transparency, repeatability, and rigor. How rigorous or thorough the search is (i.e., how many databases or languages are searched) will depend on the question being asked, the urgency of the situation, and the resources of the team involved. Of course, the more comprehensive the search is, the better it will be able to inform policy and practice. Regardless of the scope of the effort, the ultimate purpose of any review should be considered when the format is chosen, and the methods must communicate biases that can arise from less thorough search efforts. We agree with Griffin and Hayward (2019) that systematic reviews offer opportunities for engaging with stakeholders in a productive and meaningful way but that making those connections initially can be a challenge. As it becomes more of a priority for our fields to interact and communicate, our hope is that connections will be easier to forge and more of a priority to maintain. The idea of a central registry to facilitate communication between scientists and managers is certainly an exciting one. We applaud and encourage all efforts to make those connections easier. In addition, we agree with Sih et al. (2019) that formulating the systematic review question is key and that these questions should be rooted, whenever possible, in existing or emerging theoretical and conceptual frameworks. Without an understanding of mechanism, the ability of evidence to generalize across species or contexts is greatly diminished. However, stakeholder engagement still remains a crucial part of the question formulation process to ensure that the review question is not only useful, but also relevant. By bringing scientists and other stakeholders together in the question formulating process, systematic reviews can help bridge the much discussed gap between academia and the real world. Part of facilitating communication with stakeholders involves transforming the science into a digestible format. For some stakeholders, this may be the narrative synthesis alone (which, as Griffin and Hayward (2019) bring up, can be a challenge to craft in an unbiased way). For other stakeholders, the act of communicating results may be better done in person (as Caro 2019 notes), at workshops, conferences, meetings of species’ recovery groups, or via other media platforms. The fact that practitioners do not often have time to read primary literature (Caro 2019) is a major reason for engaging with the full systematic review process, not a drawback of the process itself. As scientists, it is our job to synthesize what conclusions can be drawn from the evidence base and tailor their presentation to the intended audience. By combating evidence complacency outside of our scientific bubble, we can increase the likelihood that it will be used (Walsh et al. 2014), infinitely more so than if we do nothing. Overall, the pursuit of systematic reviews is not an easy task, as several authors note. Covering highly heterogeneous fields is a challenge, but one that we hope scientists will meet. As Griffin and Hayward (2019) suggest, despite the effort involved, systematic reviews should offer a worthwhile use of academics’ time if they want their science to have meaningful impact.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.