What is Wrong With Systematic Reviews and Meta-Analyses: If You Want the Right Answer, Ask the Right Question!

We see more and more systematic reviews (qualitative pooling) and their variant meta-analyses (quantitative pooling) being published in the Aesthetic Surgery Journal. Systematic reviews and meta-analyses are on the top of the hierarchy of evidence; hence their impact in practice can be quite large if done correctly.1 Unfortunately the current quality of published systematic reviews in plastic surgery is poor.2 The statistical jargon that is presented in most systematic reviews and meta-analyses seems impressive for the naïve reader who may unwittingly accept their conclusions. In this edition of the Hub we will use the published meta-analysis by Carloni et al, “Are There Factors Predictive of Postoperative Complications in Circumferential Contouring of the Lower Trunk? A Meta-Analysis”3 as a springboard to highlight some basic concepts of systematic review and meta-analysis. We also explain some of the common pitfalls and offer guidance how to improve your next systematic review or meta-analysis (or critical appraisal thereof!). The main difference in a systematic review from a regular review is the great effort that investigators make and steps they take to avoid the introduction of bias in answering their clinical question when performing a systematic review. The same applies on the reporting of their findings. There are various steps in which bias can be introduced in the review. Table 1 below shows the key differences between a regular review and a systematic review.4 When a systematic review provides a quantitative summary and a statistical synthesis, we label this as a meta-analysis.4 Differences Between Narrative Reviews and Systematic Reviews Adapted from Cook et al.4 Differences Between Narrative Reviews and Systematic Reviews Adapted from Cook et al.4 While a systematic review is a structured approach to collect and synthesize data qualitatively to answer an important clinical question, a meta-analysis takes this pooling of information one step further; it pools and analyzes the data quantitatively. Like a systematic review, there is a defined methodology to performing a meta-analysis which is structured and regimented. This study design was introduced in the 1970s to address previously unanswerable questions. Three important reasons to perform a systematic review/meta-analysis are: To answer an important or controversial clinical question in which there is uncertainty of the superiority of one approach vs another (eg, chemical peeling vs laser resurfacing in facial rejuvenation, round vs anatomical implants in breast augmentation, or quilting sutures vs drains in abdominoplasty). To combine the results of multiple small similar studies to improve the statistical power of the study and find the true effect of the “novel” approach. To find the true effect of a novel intervention and thus identify the “boundary of knowledge” on the subject which can then lead to a new line of clinical research. When thinking about these reasons for performing a meta-analysis, it becomes clear that to find the correct answer, the authors must ask the right question … in the right way. The right kind of question for a meta-analysis is highly specific. A sound meta-analysis asks a “foreground” question as compared to a “background” question.5 A “background” question is one that asks for general knowledge about a particular clinical problem. The background question has two essential components: (1) a question root (who, what, where, when, why) with a verb; and (2) a clinical disorder. Examples of a background questions are: “Why do body contouring patients suffer high complication rates?” and “How can we prevent capsular contractures in breast augmentations?” “Foreground” questions ask about the specific management of patients with a clinical problem. A foreground question identifies five key essential components of a question: (1) the Patient Population or problem; (2) the main Intervention; (3) the Comparative Intervention (if relevant); (4) the clinical Outcome; and (5) the Time horizon. The foreground question can be remembered by the acronym PICOT. So let's say that you want to know if a belt lipectomy or a standard abdominoplasty is better. Using our PICOT acronym and defining each of the components, we can build the right kind of question that can be addressed by a systematic review/meta-analysis: So putting this all together, here's the sample question build on the above PICOT format: “In Body contouring patients who undergo belt lipectomy is there an improvement in health related quality of life at 5 years as compared to regular abdominoplasty?” This question can then be used to define the search terms, inclusion and exclusion criteria, and the interpretation within a systematic review or meta-analysis. In this way the systematic review (and, if applicable, subsequent meta-analysis) is laser-focused on answering this specific question. Just like we work hard as aesthetic surgeons to match the right patient to the right procedure, it is important in research that we have the right study design for the right question. Not all questions need to be answered with a systematic review or meta-analysis. For example, when we appraise the meta-analysis by Carloni et al, it is obvious that the investigators did a lot of work and we can see some of the tools of a systematic review/meta-analysis in action. However, the question that the authors tried to answer can be paraphrased as “What are the rates and potential predictive risk factors for complications in circumferential contouring of the lower trunk?” This is a “Background” question that could have been answered with a cohort study (observational design) alone. For sure the authors provided us with a list of complication rates in body contouring surgery which we can use to educate our next patient; however, they did not ask the kind of question best answered by a meta-analysis. In discussing the limitations of their study, Carloni et al stated that “… no definitive conclusion that is valuable from a clinical perspective can be drawn. Randomized controlled trials are needed to confirm our findings.”4 This statement is fundamentally wrong as a randomized controlled trial is not the appropriate study design to assess complications in surgery. The most appropriate study design to assess “harm” is a case-control design. To best harness the power of a meta-analysis it would have been preferable for the investigators to have asked a “foreground” question. An example of such foreground questions for this body contouring topic - based on the PICOT format - could be something like “In patients who undergo circumferential contouring of the lower trunk after massive weight loss, is buttock auto-augmentation associated with a higher rate of complications compared to circumferential contouring alone at one year follow-up?” or still better “In patients who undergo circumferential contouring of the lower trunk after massive weight loss, does buttock auto-augmentation provide better Health Related Quality of Life (HRQL) as compared to circumferential contouring alone at one year follow-up?” A meta-analysis is precisely the right study methodology to answer these types of questions and which could guide your clinical decision making. We can see that it is important for investigators to understand the merits of the various study designs; they should choose the appropriate one that will answer their specific question. There are many resources and reporting mechanisms which authors can use in the performance of a systematic review or meta-analysis. We'll talk about these in a future EBM HUB article, but a few are listed in the advice below. While these guidelines, tools, and data presentations are incredibly helpful, it is important that authors completely describe how the results were achieved. For example, although Carloni et al stated that their meta-analysis was performed in accordance to the PRISMA statement and was AMSTAR-compliant, we remain uncertain of this. Nowhere in the article do we see evidence of the inter-reviewer agreement on the selection of the studies or extraction of data from the primary studies. These are two key steps where bias can be introduced in the study. This should be reported in the results section using the Kappa statistic. This will reassure us, the readers, more than just stating that two independent reviewers screened the articles. Forest plots are also important visual representations of the data. Like matching the right study design for a given question, the right data representation needs to be matched to the question. The Carloni et al forest plots list complication rates with their confidence Intervals alone, however forest plots are best suited to depict the difference between two comparative groups. If the authors had tried to answer a “foreground” question, we would have had two such comparative groups. The Forest plots would then indicate which approach is favored or not favored. It could—for example—tell us if the addition of auto-augmentation in body contouring surgery is associated with a prohibitive complication rate or improved HRQL in a face-to-face comparison of studies that did not use it. If you are thinking about performing a meta-analysis or just getting better at your critical analysis, here are the foundations of a good systematic review and meta-analysis and a few tools to help you: Understand the difference between a regular review and a systematic review. Ask a good clinical “foreground question.” Involve key players in your research team that will tackle the question. They can include a: (i) content expert (aesthetic surgeon); (ii) librarian who will assist you with the exhaustive literature search; (iii) biostatistician; (iv) research assistant; or (v) other collaborators. Key point: Include all of them at the inception of the study and not half-way or at the end of the study! Read carefully the PRISMA statement (http://www.prisma-statement.org/PRISMAStatement/PRISMAStatement.aspx). Be familiar with the various methodological appraisal or reporting tools, such as AMSTAR. These vary depending on the study design. Be familiar with the Enhancing the Quality and Transparency of Health Research (EQUATOR) network website (www.equator-network.org) where guidelines for reporting specific research designs can be found. Examples include CONSORT, STROBE, and Transparent Reporting of Evaluations with Nonrandomized Designs (TREND). Register your study with PROSPERO (to be transparent on what you intend to do). The authors have no conflict of interests to disclose related to the content of this article. The authors received no financial support for the research, authorship, and publication of this article.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.