Factors Associated with the Reproducibility of Health Sciences Research: A Systematic Review and Evidence and Gap Map

Stephana Julia Moss,¹ Juliane Kennett,² Jeanna Parsons Leigh,³ Niklas Bobrovitz,⁴ Henry T. Stelfox⁵ <h4>Objective </h4> To map the evidence for factors (eg, research practices) associated with the reproducibility of methods and results reported in health sciences research. <h4>Design </h4> Five bibliographic databases were searched from January 2000 to May 2023, followed by supplemental searches of high-impact journals and relevant records. We included health science records of observational, interventional, or knowledge synthesis studies reporting data on factors related to research reproducibility. Factors were operationalized as modifiable or nonmodifiable aspects of study conduct relating to individual-, study-, or institutional-level practices, methods, and processes that could impact the reproducibility of research methods or results.¹ Reproducibility was operationalized by 2 mutually exclusive categories: (1) methodological reproducibility (ie, the ability to exactly repeat the methods, including study procedures and data analysis) and (2) results reproducibility (ie, obtaining corroborating results using the same or similar methods).² We included studies that used surrogate measures for reproducibility (eg, type 1 or 2 error rates) if they (1) explicitly stated their aim to investigate the reproducibility of research and (2) rationalized their choice of surrogate measure.³ Data were coded using inductive qualitative content analysis, and empirical evidence was synthesized with evidence and gap maps. Study risk of bias was assessed using the Quality in Prognostic Studies risk-of-bias tool. Statistical tests of the association between factors and reproducibility outcomes were summarized as reported in the included articles. <h4>Results </h4> Our review included 148 primarily biomedical and preclinical (n = 62) and clinical (n = 71) studies. Factors were classified into 12 modifiable (eg, sample size and power) and 3 nonmodifiable (eg, publication year) categories. Of 234 reported evaluations of factors, 76 (32%) assessed methodological reproducibility and 158 (68%) assessed results reproducibility. The most frequently reported factor was transparency and reporting (38 of 234 assessments [16%]). A total of 155 factors (66%) were evaluated for statistical associations with reproducibility outcomes (<b>Table 25-0858</b>). Statistical associations were most frequently conducted for analytical methods (24 of 26 reporting significance [92%]), sample size and power (21 of 23 reporting significance [91%]), and participant characteristics and study materials (10 of 12 reporting significance [83%]). Risk-of-bias assessments found low risk of bias for study participation, factor measurement, and statistical analysis, and high risk of bias for confounding. https://assets.underline.io/markdown_image/1/image/b16148b83c9aa6aaf32a839e2bc713d9.png <h4>Conclusions </h4> Our review identified a large body of literature consisting primarily of observational studies of factors associated with the reproducibility of health sciences research. The data suggest that reproducibility may be improved by implementing more stringent statistical testing procedures and thresholds, sample size and power calculations, and improved transparency and completeness of reporting. Experimental studies are needed to test interventions to improve reproducibility. Factors identified in this study with consistent observational support should be prioritized for experimentation. Factors that affect reproducibility in health and social care services and population and public health need to be identified given the paucity of data in these areas. <h4>References</h4> 1. Goodman SN, Fanelli D, Ioannidis JP. What does research reproducibility mean? <i>Sci Transl Med</i>. 2016;8(341):341ps12-341ps12. doi:10.1126/scitranslmed.aaf5027 2. Niven DJ, McCormick TJ, Straus SE, et al. Reproducibility of clinical research in critical care: a scoping review. <i>BMC Med</i>. 2018;16:1-12. doi:10.1186/s12916-018-1018-6 3. Clemens MA. The meaning of failed replications: a review and proposal. <i>J Econ Surveys</i>. 2017;31(1):326-342. doi:10.1111/joes.12139 ¹Faculty of Medicine, Dalhousie University, Halifax, Nova Scotia, Canada, sj.moss@dal.ca; ²Department of Critical Care Medicine, University of Calgary, Calgary, Alberta, Canada; ³Faculty of Health, Dalhousie University, Halifax, Nova Scotia, Canada; ⁴Department of Emergency Medicine, University of Calgary, Calgary, Alberta, Canada; ⁵Faculty of Medicine &amp; Dentistry, University of Alberta, Edmonton, Alberta, Canada. <h4>Conflict of Interest Disclosures</h4> None reported. <h4>Funding/Support</h4> This work was funded by the Canadian Institutes of Health Research. <h4>Role of the Funder/Sponsor </h4> The funding body had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the abstract; or decision to submit the abstract for presentation.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.