Cardiovascular meta-analyses: fool’s gold or gold for fools?

It takes many good deeds to build a good reputation, and only one bad one to lose it Benjamin Franklin Science is by definition iterative and collaborative, with the need for constant reappraisal and external validation, and cardiovascular research is no exception.1 There has been a veritable democratization of research efforts, and it is a commonplace for the cardiovascular community to witness the conduct and publication of numerous clinical studies with similar scope and goals within a short time frame. The issue in such cases is how to inform ourselves and improve decision-making. In other words, which trials should be trusted? All of them? Only the largest ones? Mainly investigator-initiated or government-funded trials? Or those of higher methodological quality (Figure 1)? Basic premises for the methodologic transition from clinical studies to meta-analyses, including network meta-analyses, and their best interpretation and application. (A) Basic approach for designing and reporting studies, appraising their internal validity, and summarizing clinical evidence. (B) Logic framework for the transition from pairwise meta-analysis to network meta-analysis. (C) Roadmap to reading and applying a meta-analysis for individual patient care. GRADE, Grading of Recommendations, Assessment, Development, and Evaluations; PICO, Patient/population, Intervention, Comparison and Outcomes; ROB-2, Cochrane Risk of Bias Tool-Version 2. Evidently, low quality studies may at best have descriptive roles, but most randomized trials are considered trustworthy, when randomization is correctly implemented (e.g. with effective allocation concealment), the hypothesis, sample size, and statistical analysis plan are pre-specified, objective endpoints are ascertained, and missing data are minimized. Yet, even limiting ourselves to randomized trials does not offer a meaningful solution, as often several trials on a specific clinical question may be reported within a few years. Meta-analysis consists in the weighted combination of several studies with distinct similarities. Analytically, the concept amounts to established statistical techniques for weighted inferences.2 Indeed, the very term meta-analysis is now 46 years old and was not devised by a clinical researcher, but a psychologist aiming to reconcile discrepant findings from the psychotherapy literature.3 Systematic reviews and meta-analyses are crucial before performing a new study (or applying for funding), and some leading journals (e.g. The Lancet) require an updated meta-analysis as a key framework to interpret an individual study’s results. In the last few decades, meta-analysis has seen major ongoing success with the advent of online search strategies (e.g. PubMed) and, most recently, with the diffusion of relatively user-friendly statistical packages and tools (see Supplementary material online). However, we are now facing a veritable overwhelming overproduction and dissemination of meta-analyses, due to many factors, including relative ease of conduct and high likelihood of scholarly citations. Accordingly, the pressing question is: are meta-analyses still useful? Are they still precious, or simply fool’s gold? The answer is not simple, and depends on several considerations. Here, we propose a roadmap to gauge the quality and usefulness of any meta-analysis. Indeed, we use a similar roadmap when reviewing meta-analyses for scholarly journals or commenting on them. First, the meta-analysis should serve a useful purpose, and inform the scientific community on areas of uncertainty by providing novel evidence. Second, preregistration on an online registry [e.g. the International Prospective Register of Systematic Reviews (PROSPERO)] is a key pre-requisite to ensure that data and application of methods are pre-specified as much as possible in order to minimize adjustment of methods to suit specific results. Third, reporting should be transparent and conform to established guidelines [i.e. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)], and clear statements along the Patient/population, Intervention, Comparison, and Outcomes (PICO) design/reporting recommendations should be able to inform the reader appropriately. Fourth, the meta-analysis should generally search, select, and abstract data from homogenous studies (e.g. randomized trials rather than observational studies for intervention meta-analyses, diagnostic test accuracy studies for diagnostic meta-analyses, and so forth). Endpoints should be explicitly defined, collected in a homogeneous (hopefully) fashion at various time points, and collected in summary tables that are reproducible. Finally, statistical analysis should be based on state of the art methods for pooling, providing uncertainty estimates (e.g. confidence or credible intervals), together with detailed results of graphical and analytical tests for statistical inconsistency and small study effects. Many excellent examples of practice changing meta-analyses exist, such as the landmark work by Hlatky and colleagues4 comparing surgical vs. percutaneous revascularization for coronary artery disease, which was wholeheartedly adopted by guidelines.5 However, and despite careful attention to all key steps, meta-analyses can occasionally provide results which are biased or at least not externally valid (i.e. not confirmed by subsequent landmark trials), such as the case of the meta-analyses on aspiration thrombectomy for ST-elevation myocardial infarction, which were not confirmed by subsequent pivotal trials (though this discrepancy could have also reflected other concomitant changes in practice).6,7 We have thus compiled a detailed series of questions and answers which may hopefully prove useful to distinguish actual gold from fool’s gold when reading a meta-analysis (Table 1, Figure 2). Case studies in meta-analysis, highlighting important methodologic and interpretation subtleties. (A) Similarities and discrepancies between estimates of relative and absolute risk effects, with two hypothetical meta-analyses having identical P-values, similar relative effect estimates, but hugely different absolute effect estimates. (B) Meta-analysis in the absence vs. presence of high risk of small study effects (including publication bias). (C) Real-world scenario of discrepant results of random vs. fixed effect modelling for pairwise meta-analysis, due to different weighting of individual studies. In similar scenarios, the most clinically conservative estimate may be considered more externally valid (in the present case, the lack of effect of magnesium in patients with acute myocardial infarction). CI, confidence interval; OR, odds ratio; RD, risk difference; RR, relative risk. Modified with permission from da Costa and Juni, Eur Heart J 2014;35:3336–3345. Key questions and pertinent answers on cardiovascular meta-analyses While such short list of question and answers may offer only superficial guidance at perusing the main contents of a meta-analysis, we find them useful as an informed starting point. Evidently, despite ongoing calls to stop meta-silliness (i.e. the plethoric and often redundant publication of meta-analyses), it is unlikely that authors will stop submitting them, journals will stop publishing them and, accordingly, readers will stop reading them. Maintaining a critical perspective is key when reading any research report, and this applies of course also to meta-analysis. Becoming informed users of meta-analysis will ensure you can distinguish gold from rubbish, and avoid ending up fooled by a flawed meta-analysis. Supplementary material is available at European Heart Journal online.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.