Need for criteria to appraise the methodological quality of laboratory‐based studies included in systematic reviews within the speciality of Endodontology

Laboratory-based studies that are not conducted in vivo on humans or animals are a major and important component of research in Endodontology. They play a critical role in the advancement of the speciality and often form the basis for the development of new endodontic therapies, instruments, devices and materials. They are employed for the standardized testing of a range of variables in well-controlled, cost-effective and repeatable environments and are usually associated with less demanding ethical implications. For example, laboratory studies can generate new hypotheses for potential clinical therapies that will be evaluated first in animals prior to potential clinical assessments in humans (Faggion, 2020). In essence, laboratory studies are ‘a building block’ within the entire hierarchical research process and form part of the translational process from bench to bedside (chairside; Drolet & Lorenzi, 2011) that aims to develop more effective treatments to alleviate the suffering and disability from human diseases, infections, malformations, trauma and conditions. As with any other type of research, laboratory-based studies should be appraised for their quality. The basic reason is simple: without tightly regulated and reproducible conditions, without methods that prevent errors, without relevant criteria, without adequate sample sizes, without standardized samples, without objective safeguards to prevent subjectivity, without adequate controls to help validate results and without accurate analysis and reporting, studies with methodological limitations can generate potentially biased and unreliable results. There is a need to use systematic reviews to compare the data between several different laboratory-based studies, to identify any outlying and presumably inaccurate results. Systematic reviews with or without meta-analyses of laboratory-based (in vitro) studies play a vital role in establishing the foundation for animal and clinical studies (Elshafay et al., 2019). The inclusion of primary studies that have a poor methodological quality within systematic reviews has a direct influence on the quality of the review in terms of the strength of the evidence and the recommendations that can be made (Shea et al., 2007, 2017). Performing an assessment of the methodological quality of individual and groups of studies is challenging, because of the substantial variability and broad scope of laboratory research. The extensive heterogeneity amongst laboratory studies can help explain why there has been a lack of widespread consensus to develop a standard approach, or validated criteria (or tool) to systematically assess the methodological quality of laboratory studies in Endodontology. However, the complex challenges of assessing the quality of laboratory studies should not be allowed to hinder efforts to develop a set of criteria to assess their quality. There is a range of methodological approaches to assess scientific studies. Some tools have been developed to assess the quality of the reporting. As example, there are guidelines developed or endorsed by the EQUATOR Network (Altman & Simera, 2016), which are required by many journals to be used as a standard process to evaluate the reporting quality of manuscripts submitted for publication. Several journals, which publish studies in the field of Endodontology, endorse some of the criteria checklists from the EQUATOR Network, and other reporting guidelines specifically developed for the speciality, for example Preferred Reporting Items for Case reports in Endodontics (PRICE) 2020 guidelines (Nagendrababu et al., 2020), Preferred Reporting Items for Laboratory studies in Endodontology (PRILE) 2021 guidelines (Nagendrababu et al., 2021). These reporting guidelines focus on how well a study is reported; however, they are not designed to assess whether a study has flaws in terms of its internal validity. For this task, tools that may assess the risk of bias (RoB) of a study have been introduced. One example is the Cochrane tool to assess the RoB of randomized controlled trials, which has been modified and improved over the years (Sterne et al., 2019). A study has a high RoB when an important methodological flaw is present that can lead to an under- or over-estimation of the true treatment effect (https://training.cochrane.org/handbook/current/chapter-08). In other words, a biased study is more likely to generate distorted results. The third concept is related to the methodological quality of studies: here, the appropriateness of the assessment depends on whether or not safeguards against bias were implemented within the methodology of the study (Furuya-Kanamori et al., 2021). The methodological approaches are usually available in the form of checklists and have been developed to cover a variety of study designs, for example the Critical Appraisals Skills Programme (CASP; https://casp-uk.net) and the Joanna Briggs Institute (JBI; https://jbi.global) checklists for clinical and non-clinical studies. Some other methodological quality checklists have a more specific focus such as A MeaSurement Tool to Assess Systematic Reviews 2 (AMSTAR-2; Shea et al., 2017) and the Appraisal of Guidelines, Research and Evaluation II (AGREE II; Brouwers et al., 2010) on systematic reviews and clinical guidelines respectively. Remarkably, there are no methodological quality assessment criteria available for scientific research, including evaluating laboratory studies in Dentistry. Generally, most criteria for evaluating the methodological quality of studies are not intended to rate their quantitative quality (Furuya-Kanamori et al., 2021), whereas uniquely, the RoB criteria aim to determine a study's quantitative ratings for its RoB (‘low risk of bias’, ‘some concerns’ or ‘high risk of bias’) at the domain and overall study levels (Sterne et al., 2019). Therefore, RoB and methodological quality tools can be similar in terms of items, but vary in their outcome applications. In other words, the numbers (count) of checklist variables in methodological quality assessment criteria can be greater than in criteria aimed solely to assess RoB. Moreover, some variables related to methodological quality, such as the sample size, might not be related to ‘risk of bias’ (RoB) but related more to the precision of the results (Viswanathan et al., 2017). The major challenge in developing new criteria to assess the RoB in clinical studies is to measure whether the methodological quality safeguards are sufficient to protect it against bias. For example, one can argue that the masking of operators in a randomized clinical trial to compare surgical with non-surgical endodontic treatment is technically impossible, because the procedures are so obviously different, and these differences are identifiable to the clinician. The researchers could have taken all the possible assessment measures (i.e. utilizing methodological quality safeguards) to undertake a study with a high quality, but masking in this specific case, is fundamentally impossible, so arises the controversy, because a randomized clinical trial can be considered to have high quality, but at the same time, also have a high RoB. This is because the lack of blinding can generate some potential bias in relation to the outcome assessed. In this scenario, the high RoB is obviously not an intentional fault of the researchers; however, the high RoB means that a detailed and in-depth interpretation of the data is necessary to understand whether this high RoB could distort the study outcomes. For laboratory studies, it would be even more challenging to determine quantitatively whether any specific procedures would be capable of distorting the effect size estimate and RoB due to the substantial variability amongst treatments/procedures, devices and/or materials as well as their analysis and reporting outcomes. The most realistic assessment of laboratory studies in Endodontology might be best achieved through developing a comprehensive analysis of quality criteria to evaluate the methodological quality of these studies. The main advantage of approaches aimed to evaluate the methodological quality is the greater opportunity to discover variables that not only might be related directly to a high RoB, but also might be difficult to prove beyond any reasonable doubt whether or not they distorted the results of the study. Such factors would include, for example, a subconscious or conscious RoB created by the ‘conflict of interest’ of the researchers conducting the study. Moreover, a checklist of methodological criteria is essential to allow readers to understand and interpret the outcomes and RoB that is presented to them from the manuscript. Furthermore, by observing these criteria to assess their methodological quality, researchers will benefit from greater opportunities to increase the accuracy, reliability and reproducibility of laboratory research, which is an essential foundation of good science and a measure of research integrity (Resnik & Shamoo, 2017). The purpose of this editorial was to highlight the challenges of assessing the methodological quality of laboratory studies and to emphasize that work on developing methodological quality criteria to assess laboratory studies in Endodontology has begun. The criteria will be developed by a formal validated consensus process involving a steering committee made up of methodologists, clinicians, academics and statisticians. These criteria will be developed using rigid methodological principles, and by observing detailed chronological steps, to ensure that they reflect what the criteria are intended to accomplish, that is to determine the methodological quality of laboratory-based studies in Endodontology, that is research not conducted in vivo on humans or animals. It is anticipated that researchers, including those that undertake systematic reviews of laboratory-based studies and readers of the literature in Endodontology, will benefit by adopting these criteria and applying them to this important area of research. The authors deny any conflicts of interest related to this editorial. CF, VN and PD involved in the conception of the editorial. VN, PA, CB, HD, CF, AK, PM, SP and PD drafted the first version of the manuscript. VN, PA, CB, HD, CF, AK, PM, SP and PD were involved in the final approval of the manuscript. The editorial did not require ethical approval.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.