Caution should be exercised when assessing ivermectin for the treatment of COVID‐19 in systematic reviews

Dear Editor, We read with great interest the systematic review and meta-analysis by Hariyanto et al. on the efficacy of ivermectin treatment across COVID-19 patients.1 The authors concluded that ivermectin may offer beneficial effects after noting a significant reduction associated with ivermectin in outcomes such as illness severity, mortality and time to hospital discharge. We greatly appreciate such insights on this topic which has received considerable attention as COVID-19 management strategies evolve and individuals continue to require hospitalization. At the same time, there are several limitations in the methodology and conduct of this review that must be considered to ensure appropriate interpretation of the findings. First, to ensure quality and reproducibility, systematic reviews should be conducted and reported according to the latest Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)2 framework and Cochrane guidelines.3 In particular, risk of bias and quality of evidence assessments are integral to understanding the strength of the meta-analysis results. In this systematic review, the authors utilized the Jadad scale4 to assess study quality; however, it is unclear whether this tool was utilized correctly or provided a valid assessment. For example, while open-label studies should have been given zero points for blinding, the authors incorrectly gave one point for two open-label studies.5, 6 If these studies were assessed appropriately using the Jadad scale, it would have downgraded their moderate-quality ratings to low. Additionally, while Elgazzar et al.7 was rated as having moderate quality, it has since been retracted from its preprint server due to poor study conduct. This raises concerns that the Jadad scale may not have been a valid and reliable indicator of study quality in this review. Using robust risk of bias tools—such as those developed by the Cochrane Collaboration (RoB and RoB2)8, 9—may have been more appropriate, especially as most studies rated as having good methodological quality by the Jadad scale were found to have a high risk of bias according to RoB.10 Moreover, the authors did not provide a Grading of Recommendations, Assessment, Development and Evaluation (GRADE)11 assessment on the confidence of their findings as required on the latest PRISMA 2020 guidelines2; therefore, the overall certainty of the study findings cannot be evaluated. Second, while the authors stated that missing mean and standard deviation values were imputed from medians and interquartile ranges according to the method proposed by Wan et al.,12 it should be mentioned that non-parametric statistics are usually reported in studies due to the presence of skewness in the distribution.3 Therefore, the authors should have performed tests for normality using established methods, such as those recommended by Shi et al.,13 prior to proceeding with the imputation. Otherwise, the inclusion of potentially skewed study data in a meta-analysis, which assumes a normal distribution, can produce misleading results.13 For instance, the authors inappropriately utilized mean imputation for the time to symptom alleviation reported by López-Medina et al.14 According to the methods by Shi et al.,13 the datasets for both the ivermectin and control arms in this trial were significantly skewed from normal. Therefore, findings relating to the time to symptom alleviation reported by this trial should not have been imputed but rather narratively described. Lastly, while the authors included patient-important outcomes such as the duration of hospitalization and incidence of symptom alleviation, these outcomes were not outlined on their PROSPERO registration. Similarly, the authors performed subgroup analyses by disease severity in the final publication despite no mentions of subgroup analyses in their registration. In addition, they included a study by Shouman et al.15 that utilized ivermectin as a prophylactic regimen, deviating from their prospective inclusion criteria; although it must be noted that data from this study was not included in the meta-analyses. Conversely, the authors failed to include a relevant trial by Chaccour et al.16 which they cited in their Discussion section. The aforementioned trial clearly satisfied the authors' prospective inclusion criteria and reported relevant outcomes such as incidence of RT-PCR conversion and incidence of progression to severe COVID-19. According to PROSPERO, the outcome selection, eligibility criteria and statistical analysis plans needed to be finalized prior to data extraction to ensure the transparency and legitimacy of the systematic review process.17 Therefore, the authors should have marked any deviations from their PROSPERO registration as post hoc modifications. In addition to the above methodological limitations, we would like to also express concerns regarding the validity of the authors' data sources. Most notably, 9 out of 18 of the included studies (excluding Shouman et al.) were pre-print articles. While including pre-print articles could reduce potential publication bias,18 special care must be taken when reviewing such articles involving ivermectin. Since the introduction of ivermectin as a candidate COVID-19 treatment regimen, there has been an influx of clinical trials surrounding this drug across non-peer-reviewed sources. As pre-prints and other non-peer-reviewed studies may be published without following the recommended processes to ensure quality standards,19 it is unknown whether these publications presented results from valid trials and they have caused considerable confusion among both the general public and clinicians regarding the true efficacy of ivermectin. Case in point, the pre-print article by Elgazzar et al.7 was retracted by its pre-print server due to concerns regarding its study conducts, and this retraction had since led to a subsequent retraction of a corresponding meta-analysis.20 Therefore, it would be ideal if the authors perform an updated analysis excluding Elgazzar et al., as well as conduct sensitivity analyses excluding pre-print articles to examine the impact of including non-peer-reviewed articles in their meta-analyses. As shown in another review which only included peer-reviewed studies, ivermectin was not associated with significant improvements in patient outcomes.21 Furthermore, we found that two of the included studies22, 23 were published in potentially predatory journals based on searches in Cabell's Predatory Reports24 and Beall's list.25 Predatory journals are entities driven by self-interests that typically publish research articles without a rigorous peer-review process nor good editorial practice or transparency.26 Therefore, in the context of ivermectin, articles from predatory sources could be highly unreliable. While both the Cabell's Predatory Reports and Beall's list are not definitive nor comprehensive, the authors should consider examining questionable articles using predatory journal criteria established by previous journalology studies, such as the checklist by Richtig et al.,27 and perform sensitivity analyses excluding these studies, if necessary. The authors supported their findings by arguing that ivermectin functions against COVID-19 via two main mechanisms: (1) by directly inhibiting viral replication as evidenced by an in vitro study from Caly et al.,28 and (2) by reducing inflammation in patients experiencing hyperinflammatory responses. While Caly et al. has been cited in various reviews and publications supporting the use of ivermectin, it must be noted that it is impossible to achieve the drug concentration used by Caly et al. in vivo due to protein binding and poor penetration of ivermectin into the lungs, even at the highest reported ivermectin dosage of 1700 μg/kg.19, 29 Thus, to replicate the beneficial effects observed in vitro, a dangerously high dosage of ivermectin may be required. Second, the theory that ivermectin functions by suppressing inflammation suggests that the timing of ivermectin would be crucial for improving patient outcomes, similar to corticosteroids.30 For instance, in patients with early-stages of the disease without immunopathologies, ivermectin may do more harm than good due to its immunosuppressive effects. While the authors collected time to ivermectin administration as a part of their study, these data were not reported nor analysed in the review. We recommend that the authors conduct a meta-regression analysis to identify potential correlations between time to ivermectin administration and treatment outcomes in order to fully elucidate the role of ivermectin in COVID-19 treatment. This research did not receive any external funding. Jiawen Deng, Fangwen Zhou and Kiyan Heybati have previously published a systematic review on a similar topic with contradictory results21 compared to the findings of the critiqued article. Kyra Kavanagh declares no conflict of interest. Not applicable. Fangwen Zhou and Kiyan Heybati analyzed the methodological design of the original article. Jiawen Deng and Kyra Kavanagh examined the data sources of the original article and performed journalology research. Jiawen Deng, Fangwen Zhou and Kiyan Heybati contributed equally to the drafting of the manuscript. All authors had revised the manuscript critically for important intellectual content, and gave final approval for the manuscript to be published. All authors agree to be held accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. No data is disseminated through this publication.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.