Mastering Meta‐Analysis

A systematic review is a comprehensive process that involves gathering all relevant studies on a specific topic and research design, followed by a thorough evaluation and analysis of their findings (Ahn & Kang, 2018). They have been conducted since 1753 when James Lind published a paper that aimed to provide a concise and impartial summary of evidence on scurvy. However, it wasn’t until after the 1970s that systematic reviews began to receive greater attention, highlighting the importance of synthesizing research findings (Clarke & Chalmers, 2018). A systematic review follows a predefined protocol, including clear inclusion criteria and search strategies (Paul & Leibovici, 2014). Systematic reviews can be divided into: (i) qualitative: if they provide a synthesis of research studies; or (ii) quantitative: if they involve the processing of a dataset gathered from previous publications (Philibert et al., 2012). Generally, quantitative systematic reviews are referred to as “meta-analyses,” when a statistical method is applied to a dataset derived from a systematic review. While all meta-analyses should be based on systematic reviews, not all systematic reviews include meta-analyses (Ryś et al., 2009). In other words, systematic reviews provide a comprehensive overview of available evidence on a topic, whereas meta-analyses offer a quantitative summary of effect sizes (Table 1). Both methods are valuable for supporting clinical decision-making and developing evidence-based guidelines, and they require careful interpretation and critical evaluation. A systematic review should always begin with identifying a research question that is clear, interesting, and unique. In fact, having a well-defined research question sets a systematic review apart from a general literature review while striving to answer a quantitative question turns it into a meta-analysis. There are many approaches to formulating a research question, for instance—authors can use a FINER (Feasible, Interesting, Novel, Ethical, Relevant) criteria or PICO (Population, Intervention, Comparison, Outcome) or SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) framework to formulate a clear and focused research question. Researchers suggest a combined approach, utilizing either or both the SPIDER and PICO tools, to achieve a comprehensive search, depending on the available time and resources (Tawfik et al., 2019). Here’s an example of a research question authors may ask: “What is the effect of organic mulch application (Intervention) compared with inorganic mulch (Comparison) on soil moisture retention (Outcome) in arid regions (Population)?” Feel ready to formulate a research question? Look up in depth on FINER (https://tinyurl.com/finercriteria) criteria and PICO (https://www.cochranelibrary.com/about-pico) through the links. Once authors have identified a research question that is novel and feasible for study, the next step is to delineate the boundary (scope) of the study. This can be achieved through establishing specific criteria for including or excluding certain kinds of studies in the review to ensure consistency of research and relevance in applicability of findings. For instance, in the above example, authors can decide to only include peer-reviewed journal articles that report field studies with a minimum of two years or more, or studies that report the effect on both a soil and crop parameter (or a particular parameter), or studies conducted in a particular type of soil (e.g., sandy) or climatic regions (e.g., temperate). Then, the next step is the tedious act of going through hundreds of titles and abstracts to identify the studies that meet the inclusion/exclusion criteria that the authors devise. There are different software or websites that can help to organize this step, for example HUBMETA (https://hubmeta.com/). After these steps, it is a good idea to formulate a tentative title of the review, develop a protocol, and register the title and protocol with platforms like OSF (https://osf.io/) or PROSPERO (https://www.crd.york.ac.uk/prospero/) etc. to ensure transparency. These platforms allow authors to add titles, author name(s), objectives, inclusion/exclusion criteria, search strategy, data extraction methods, and statistical analysis plan. Then, authors should conduct a comprehensive literature search across multiple databases and sources, document the selection process, and assess the risk of bias in each study to further identify the studies that can be trusted and used. It involves finding relevant literature databases, picking suitable keywords, and employing search filters (Langenfeld & Singh, 2024). The common literature databases are Google Scholar, Scopus, Web of Science, PubMed, and ResearchGate. The screening process should be well documented and reproducible, often guided by tools like PRISMA flow diagrams (https://www.prisma-statement.org/prisma-2020-flow-diagram). At this stage, if the aim is to conduct a meta-analysis, authors should make sure that either the data presented in the paper are sufficient, are available online, or can be collected from the authors for the study. For a narrative systematic review, this may not be a requirement. In the previous example, authors can combine key words such as “biochar,” “soil carbon,” “crop yield,” and “field trials.” This involves a rigorous step of reading full papers once the eligibility is confirmed to extract information or data (for meta-analysis) required to answer the research question. In each step, it is key to document the search strategy in detail, including databases searched, key words used, and date of the search, especially when working on a collaborative project. For extracting data, it is recommended to use standardized forms to extract data on study characteristics, methodology, outcomes, and quality. When such information is not readily available, authors can be contacted to gather more information. Study characteristics may include publication metadata (source, date, authors, peer-review status), study design (randomized, observational, cross-sectional, split, etc.), study eligibility criteria (PICO), study implementation (intervention type, length of follow up, study population, number of samples, etc.), participant information (average age, sex, demographics, etc.), and more. On the other hand, outcomes information can include outcome measures and effect sizes (mean, median, range, standard deviation, [standardized] treatment difference, correlations), sample sizes (total, per group, stratified by participant characteristics, enrolled, lost to follow up, analyzed), statistical inference (standard errors [SE], confidence intervals, p-values), and analysis methods (statistical model, statistical test, estimation method, assumptions). Use of PRISMA tool to identify, screen, and select the peer-reviewed articles to conduct a systematic review and meta-analysis. Image by Page et al. (2021) and reprinted here under this license: https://creativecommons.org/licenses/by/4.0/. For instance, in the running example, it can be crucial to note down the substrate of organic mulch used, its composition, physical and chemical characteristics, application method etc. (study characteristics) as well as parameters such as changes in soil physical and biological properties and impact of crop yield (outcome of interest information). For meta-analysis, selecting appropriate effect size measures and statistical models (e.g., fixed-effect or random-effects models) is key for data synthesis. Depending on the outcome type (continuous, ordinal, dichotomous, counts, etc.), appropriate summary statistics should be selected (Table 2). To conduct a multivariate analysis, it can be a good idea to perform subgroup analyses and meta-regression to explore heterogeneity and potential moderators of effect. Meta-analysis can be conducted with any statistical software that allows random effect models. Most meta-analysis methods employ a variation on a weighted average of the effect estimates from the different studies. Meta-analysis-specific options provide commonly reported statistics and graphics R packages (dmetar, forester, meta, metadata, metafor, etc.), STATA (meta, many others from community), RevMan (specifically for Cochrane reviews), SPSS, SAS (PROC, MIXED, PROC NLMIXED, macro, etc.) It can be crucial to conduct a heterogeneity and sensitivity analysis, which can be accomplished through: assessing heterogeneity (use the I2 statistic and Q test to assess variability among study results), subgroup analysis (conduct subgroup analyses to explore potential sources of heterogeneity) and sensitivity analysis (perform sensitivity analyses to examine the robustness of the results). Authors can use tools like the Cochrane Risk of Bias tool for randomized controlled trials or the Newcastle-Ottawa Scale for observational studies. Using tools such as funnel plots (scatter plot of study effect estimates vs. precision [SE, sample size]), authors can identify and address outliers that may disproportionately influence the results. After gathering all the necessary information and summarizing the key findings, the next step is finalizing the story and presenting/interpreting the results following guidelines like PRISMA, ensuring transparency and reproducibility in the reporting of the systematic review and meta-analysis (Table 3). It is also key to identify and address potential biases (e.g., publication bias, small study bias) and outliers in the data and report accordingly. It is a good idea to reference some exemplary systematic reviews/meta-analyses including those published in high-impact journals such as the Cochrane Database of Systematic Reviews (Deeks et al., 2023), which adhere to rigorous standards and provide valuable insights into specific research questions. Steps to conduct a structured systematic review and/or meta-analysis. How to ensure quality of systematic review and meta-analysis. Systematic reviews and meta-analyses are powerful tools for synthesizing research evidence, offering increased precision and power to evaluate interventions and relationships across studies. Adhering to rigorous methodologies and transparent reporting standards ensures the reliability and validity of their findings, making them essential for evidence-based practice and policymaking. Despite their limitations, when conducted properly, they provide invaluable insights and guide informed decision-making in various fields of research. If you would like to give us feedback on our work or want to volunteer to join the committee to help plan any of our activities, please reach out to Ariana Lazo ([email protected]), the 2024 Chair of the committee! If you would like to stay up to date with our committee, learn more about our work, contribute to one of our CSA News articles or suggest activities you would like us to promote, watch your emails, connect with us on Twitter (@ACSGradStudents) and Facebook (ACS.gradstudents), or visit: agronomy.org/membership/committees/view/ACS238/members, crops.org/membership/committees/view/ACS238/members, or soils.org/membership/committees/view/ACS238/members.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.