Why the Cochrane Risk of Bias Tool Should not Include Funding Source as a Standard Item

GP Kidd/Science Photo Library This editorial is based on a presentation given as part of a debate held at the Cochrane Methods Symposium in Québec City, Canada, October 2013. You can also read Lisa Bero's editorial putting the case for including funding source in the risk of bias tool. Since its introduction in 2008, The Cochrane Collaboration's risk of bias (RoB) tool has been used in hundreds of Cochrane and non‐Cochrane systematic reviews. Its development was based on a combination of empirical and theoretical considerations, leading to a focus on risk of bias rather than study 'quality' and a division of assessments into six bias domains.[1] Despite the substantially increased workload for review authors compared with previous approaches, this major change to Cochrane Review methodology has been widely accepted. The Collaboration is a diverse organisation, and a number of review authors and groups have freely adapted the RoB tool. Some adaptations, such as including an assessment of whether sample size calculations were reported, were explicitly rejected in the Cochrane Handbook chapter describing the RoB tool.[2] Others, such as including assessment of risk of bias due to early stopping, are the subject of ongoing debate and disagreement. Including the funding source as a standard item in the tool is an adaptation that has caused particular controversy. There is widespread agreement about the importance of knowing who funded a study. The way that the medical research community addresses and documents conflicts of interest has been transformed over the last decade, and most journals now require detailed declarations, including a stipulation to err on the side of complete disclosure, to be completed by each author at the time a paper is submitted. These innovations are based on well‐documented controversies such as the series of events leading to the withdrawal of the COX‐2 inhibitor rofecoxib (Vioxx), and more recently the resistance from Roche to full release of data from trials of oseltamavir (Tamiflu), despite heroic efforts by Chris Del Mar, his colleagues in the Cochrane Acute Respiratory Infections Group, and others.[3] More than 61,000 individuals have signed the AllTrials petition (www.alltrials.net), and more than 400 organisations (including GlaxoSmithKline and about 140 patient groups) have signed up to support the campaign for all trials to be registered and all results to be reported.[4] However, the problem with including funding source in the risk of bias tool and (I assume) classifying all pharmaceutical industry‐funded studies as at high risk of bias, is that there is little evidence (so far that I am aware) that trial methods are more likely to be flawed if a trial is industry‐funded. Nor am I aware of evidence that fraud is more likely if a trial is industry‐funded. Indeed, industry‐funded trials tend to be well‐resourced, done by highly skilled and experienced professionals, and based on detailed and extensively documented standardised operating procedures. In my opinion the problems associated with industry‐funded trials lie mainly in two areas: selective reporting of outcomes and whole studies, and problematic choice of comparator. Unfortunately, assessment of selective reporting in the current RoB tool does not work well. Many review authors classify this item as 'unclear', reliability is low, and most importantly it makes little sense to classify all findings from a study as biased on the basis that it failed to report one or more particular results (eg, relating to an adverse effect). In such circumstances the risk of bias is in the meta‐analysis examining the adverse effect rather than the study's contribution to the outcomes on which it does report. The issue is more akin to traditional publication bias, when eligible studies remain unpublished for reasons related to their results. The Collaboration needs a better‐structured approach to assessment of the risk that a meta‐analysis is affected by reporting biases, which should be completely or mainly separated from the RoB tool. Associated guidance should focus on searching for, identifying and accessing unpublished information; and include information on how to access and use trial protocols, trial registries and information available to regulators. Resulting risk of bias assessments would be at the level of the meta‐analysis (for an outcome), not at the level of an individual trial. The focus of the RoB tool should remain on ways in which a trial's methods render its results at risk of bias. There is no doubt that choice of comparator in industry‐funded trials can pose a major problem to medical decision‐makers. There is an obvious incentive for those with a commercial interest in the success of a new drug to plan a comparison that is likely to show it in a favourable light and facilitate regulatory approval and marketing campaigns. Too often, a new drug is compared with placebo or an inadequate alternative when the question of interest is whether it is better than its major competitor. However, the results of a well‐conducted and fully reported trial with inappropriate comparator are not at risk of bias – they are simply of limited utility to those wishing to make informed medical decisions. Unhelpful or inappropriate choices of comparator should be addressed through sensible interpretations of pairwise meta‐analyses, routine use of mixed treatment comparisons (network meta‐analyses) that focus on the question "What is the best intervention from among all the candidates?", and increased use of methods that integrate effects on benefits and harms in order to facilitate informed treatment choices. Transparent and structured assessments of conflict of interest in included trials should be a routine component of Cochrane systematic reviews. Such assessments relate mainly to the context in which the review results should be interpreted. It is vital to know whether most of the information comes from a company with a commercial interest in the intervention of interest, but this is not in itself a reason to dismiss the accumulated evidence. It may nonetheless be a reason to search particularly hard for unreported or selectively reported evidence and to carefully scrutinise the chosen comparators. In summary, conflict of interest in reporting of medical research is a substantial and unresolved problem. Welcome progress has been made through better regulation of pharmaceutical industry interactions with the medical profession, increased use of online trial registries, better journal publication policies, and campaigns to make all trial results accessible. However, the suppression of trial results remains a major problem that is far from being solved. There are particular problems associated with pharmaceutical industry‐funded research, but these should be dealt with by: (1) reporting and commenting on conflicts of interest as a standard component of Cochrane systematic reviews; (2) better procedures and an improved tool to assess reporting biases; and (3) more extensive use of mixed treatment comparisons. Adding source of funding as a bias domain in the risk of bias tool would send an extremely negative message to pharmaceutical industry colleagues with whom we should be happy to work, and it might have the unintended consequence of labelling high‐quality trials as biased while diverting attention from appropriate solutions to problems associated with pharmaceutical industry trials.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.