[V] Prevalence of Reproducible Health Sciences Research: A Systematic Review and Meta-Analysis
Bibliographic record
Abstract
Niklas Bobrovitz,1,2 Harriet Ware,2 Corson Johnstone,3 Juliane Kennett,3 Stephana Julia Moss,4 Liam Whalen-Browne,3 Faizan Khan,5 Benjamin Fletcher,2 Daniel J. Niven,3,6,7 Henry T. Stelfox3,6,8 Objective To synthesize evidence on the prevalence and measurement of reproducibility in health sciences research. Design This was a systematic review with meta-analysis registered a priori.1 We searched MEDLINE, Embase, Web of Science, the Cochrane Controlled Trials Registry and Database of Systematic Reviews, and grey literature published from January 1, 2000, to May 31, 2023. We included English-language articles in the health sciences that reported the prevalence of reproducible research or the prevalence of studies dedicated to reproducing research. We included studies published after 2000 to ensure relevance to current practice. Two authors independently screened articles in duplicate. Extractions were completed by one reviewer and verified by a second. The prevalence of reproducible research was defined as the proportion of research findings or studies for which the methods could be exactly repeated (methodological), the results could be corroborated by a study using the same or similar methods (results), or qualitatively similar conclusions could be made based on a study reproduction (inferential). Risk of bias was assessed using a modified Joanna Briggs Institute checklist. We meta-analyzed estimates using random effects models. To explore between-study heterogeneity, we conducted univariable and multivariable meta-regression using generalized linear mixed models. The level of significance was set at P < .05. Results A total of 177 studies published between 2001 and 2023 were included, covering all health sciences domains: biomedical/preclinical (n = 56), clinical (n = 106), health/social care services (n = 2), population health (n = 5), and multiple domains (n = 8). Studies reported on 3 types of reproducibility: methodological (n = 68), results (n = 129), and inferential (n = 5). There were 7 methodological approaches for conducting reproducibility research and 38 metrics to quantify reproducibility. The pooled estimated prevalence of reproducible research was 36.2% (95% CI, 29.7%-43.3%; I2 = 99.8%; n = 180 estimates) with variation by research domain: 30.3% (95% CI, 20.3%-42.6%; n = 59 estimates) for biomedical/preclinical, 38.3% (95% CI, 30.0%-47.4%; n = 106 estimates) for clinical, 66.6% (95% CI, 20.7%-93.9%; n = 3 estimates) for health/social care services, and 36.5% (95% CI, 0.0%-100.0%; n = 3 estimates) for population health. There was a nonsignificant decline over time between 2001 and 2023 of −4.7% (95% CI −10.1 to 1.1%; P = .11) per year in the odds of reproducibility (Figure 25-1041). The pooled estimated prevalence of studies dedicated to reproducing research was 1.4% (95% CI, 0.40%-4.10%; I2 = 99.2%; n = 40 estimates). Most estimates were at moderate risk of bias (n = 193), with some at high (n = 25) or low (n = 2) risk. Meta-regression of publication year, research domain, type of reproducibility, metric to quantify reproducibility, and risk of bias explained minimal estimate heterogeneity. https://assets.underline.io/markdown_image/1/image/58c52ca1e8ab42d080707ad8bf3acf01.png Conclusions We found that an estimated one-third of health sciences research was reproducible and 1 in 100 studies were dedicated to reproducing research. Further investigation is needed to understand heterogeneity in estimates. Standardized strategies for quantifying reproducibility should be developed. Reproducibility studies are needed in health/social care services and population health research. Reference 1. Bobrovitz N. The reproducibility of health research. Open Science Framework. 2021. osf.io/3fhd9 1Department of Emergency Medicine, Cumming School of Medicine, University of Calgary, Canada, njhbobro@ucalgary.ca; 2Centre for Health Informatics, Cumming School of Medicine, University of Calgary, Canada; 3Department of Critical Care Medicine, Cumming School of Medicine, University of Calgary, Canada; 4Department of Pediatrics, Division of Infectious Diseases, IWK Health Centre, Dalhousie University, Canada; 5Hotchkiss Brain Institute, Cumming School of Medicine, University of Calgary, Canada; 6O’Brien Institute for Public Health, University of Calgary, Canada; 7Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, Canada; 8Faculty of Medicine and Dentistry, University of Alberta, Canada. Conflict of Interest Disclosures None reported. Funding/Support The work was funded by the Canadian Institutes of Health Research. Role of the Funder/Sponsor The funding body had no role in the design or conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the abstract; or decision to submit the abstract for presentation.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.212 | 0.592 |
| Meta-epidemiology (narrow) | 0.002 | 0.002 |
| Meta-epidemiology (broad) | 0.010 | 0.020 |
| Bibliometrics | 0.026 | 0.031 |
| Science and technology studies | 0.002 | 0.003 |
| Scholarly communication | 0.009 | 0.006 |
| Open science | 0.004 | 0.004 |
| Research integrity | 0.005 | 0.003 |
| Insufficient payload (model declined to judge) | 0.005 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".