Data mining of effect sizes from PubMed abstracts: a cross-study conceptual replication

In their recent article published in Bioinformatics and entitled ‘Algorithmic identification of discrepancies between published ratios and their reported confidence intervals and P-values’ [designated thereafter by ‘Error Study’], Georgescu and Wren (2017) used a data mining approach to quantify the error rate in reported Effect Sizes (ESs) [i.e. Odds ratio (OR), Relative risk (RR) and Hazard ratio (HR)], with their 95% Confidence Interval (CI), in MEDLINE abstracts. They found a published error frequency of between 5.6 and 7.5% in extracted ESs. In the same month of December 2017, completely independently, we published an article in GigaScience entitled ‘The intriguing evolution of effect sizes in biomedical research over time: smaller but more often statistically significant’ [designated thereafter by ‘Effect Sizes Study’] (Monsarrat and Vergnes, 2018). The main objective of our study was to analyze the temporal evolution of ESs in terms of strength and statistical significance. We found a decrease of ES values in PubMed abstracts between 1990 and 2015 while, concomitantly, they became more often statistically significant. We called this phenomenon the in-silico effect. The two articles did not have the same research objectives, but they share some notable similarities. Firstly, they both use a data mining method on the same database (MEDLINE via PubMed) to extract the same entities (ESs: OR, RR or HR). The availability of the whole PubMed database via the FTP service of the NCBI allows standardized information to be collected at the abstract level, such as the abstract full text, the date of publication and the PubMed Identifier (PMID). Secondly, both articles cover a similar period, from 1990 to 2015–2016, for reasons related to the digital availability of abstracts that was set up in the PubMed database in 1996. Finally, both articles make their entire database freely downloadable to readers for transparency and reproducibility of analyses. As a result, the two final databases have similarities that can be used for cross-replication purposes. In biomedical research, replication is essential for the reliability of knowledge (Begley and Ioannidis, 2015), with obvious importance in pharmacology research, for example (Curtis and Abernethy, 2015). This is even more true in computational research, because ‘new tools and technologies, massive amounts of data, interdisciplinary approaches, and the complexity of the questions being asked are complicating replication efforts’ (Sandve et al., 2013). There are two main forms of replication: direct replication and conceptual replication (Picho et al., 2016). Direct replications are useful because they check for lack of internal validity but they are limited by the fact that any methodological flaws in the original study will be perpetuated in subsequent studies. On the other hand, conceptual replications only require that the essential conditions closely match those in the original, while allowing the flexibility to vary other non-essential conditions (Picho et al., 2016). Here, the data mining methods used to automatically extract ESs from PubMed abstracts differ between the two studies, because they were independently created by their respective authors. There is no validated process for doing this and, as far as we know, these are the first two attempts at automated extraction of ESs in the PubMed database. However, at that crucial step, it is of tremendous importance for scientific rigour that the data mining process should not influence results. In this letter, we therefore present a cross-study conceptual replication: we analyzed the final database of each study according to the main objective of the other. Thus, we first quantified the error rate in reported Effect Sizes (ESs) from the database of the Effect Sizes Study and then we assessed the temporal evolution of ESs, in terms of strength and statistical significance, from the database of the Error Study. Backgrounds, objectives, methods and databases are described in the original articles (Georgescu and Wren, 2017; Monsarrat and Vergnes, 2018). The main difference between the two databases is that the Effect Sizes Study deals with a total of 814 120 extracted ESs (OR, RR or HR), versus 244 750 for the Error Study, with an overlap of 96.9% from the first to the second. The reason for such a difference lies in the data mining process, which was much more developed in the Effect Sizes Study because of the elaboration of a complex list of admitted expressions (e.g. ‘RR,’ ‘OR,’ ‘HR,’ ‘relative risk,’ ‘odds ratio,’ ‘hazard ratio,’ ‘aRR,’ ‘aOR,’ ‘aHR’, etc.), along with additional algorithmic management of polysemy of acronyms (e.g. ‘respiratory rate’ for RR, ‘ovulation rate’ for OR, ‘heart rate’ for HR, etc.) to minimize false positive rates. A specificity of greater than 99.9% was thus achieved without loss of sensitivity (95%). Conceptual replication #1—Objective of the Error Study/Database of the Effect Sizes Study To prepare the conceptual replication and to verify our ability to reproduce the Error Study’s results, we first performed a direct replication of the Error Study, using its own raw data. We used the formulas provided in paragraph 2.5 of Georgescu and Wren’s article to re-calculate reported ES values (based on their 95% CI) and obtained similar values to those of the original article (results not shown). We then applied the same methods to the Effect Sizes Study’s database. We found similar results concerning discrepancy rates (Table 1, Fig. 1), with a range of errors estimated between 5.3 and 8.1% of reported ESs in PubMed abstracts. We also found similar results concerning the decrease in error rates over time (Fig. 2A and B), with a very recent trend (in about the last 7–8 years) for a slight re-increase of error rates over time—at least for OR and HR errors (Fig. 2B). Reported ES values versus recalculated values across order-of-magnitude discrepancy ranges for each of the item types analyzed Note: The 803 225 ESs (286 967 abstracts) excluded ESs from systematic reviews or ESs without 95% confidence intervals. Contrary to the initial analysis provided in the Effect Sizes Study (Monsarrat and Vergnes, 2018), the values before 1990 and after 2016 were not excluded. Numbers in italics in brackets correspond to original results found in the Error Study (Georgescu and Wren, 2017). Reported ES values versus recalculated values across order-of-magnitude discrepancy ranges for each of the item types analyzed Note: The 803 225 ESs (286 967 abstracts) excluded ESs from systematic reviews or ESs without 95% confidence intervals. Contrary to the initial analysis provided in the Effect Sizes Study (Monsarrat and Vergnes, 2018), the values before 1990 and after 2016 were not excluded. Numbers in italics in brackets correspond to original results found in the Error Study (Georgescu and Wren, 2017). Scatterplot of reported versus recalculated ESs (i.e. OR, HR, RR) based upon their reported confidence interval in log10 scale. Density plots are shown, reflecting the density of observations within that range of reported values (N = 803 225) Discrepancy rates between reported versus recalculated ESs are decreasing over time. (A) Scatter plot of the temporal evolution of monthly 1% discrepancy rates. For each ES, a discrepancy was considered if the error rate between reported and recalculated ESs was greater than or equal to 1%. The temporal evolution is decreasing, with a τ value of −0.40 (P < 0.001). One outlier is not shown (higher than 0.2) but was included for τ value computation. Number of ESs through the 1990–2016 period: 799 005. Reduction in variability over time may be due to increasing number of abstracts reporting ESs (more details in the Effect Sizes Study). (B) Scatter plot of the temporal evolution of monthly 1% discrepancy rates by type of ESs. The temporal evolution is decreasing for OR and RR values but increasing for HR: τ values of -0.28 (P < 0.001), −0.10 (P = 0.01) and +0.11 (P = 0.005), respectively. Fourteen outliers are not shown (higher than 0.2) but were included for τ value computations. Number of ESs through the 1990–2016 period: 520 126; 107 530 and 171 349 for OR, RR and HR, respectively Conceptual replication #2—Objective of the Effect Sizes Study/Database of the Error Study Additionally, we replicated the main analysis of the Effect Sizes Study using the raw database provided by the Error Study. We also identified the in-silico effect in this database, with a remarkable decrease of ES values in PubMed abstracts between 1990 and 2016 (Fig. 3A), concomitantly with results that become more often statistically significant (Fig. 3B). ESs are decreasing over time and proportion of statistically significant ESs has increased with time. (A) Scatter plot of the temporal evolution of yearly medians of ESs, on a positive linear scale. ESs were considered at the abstract level (N = 243 832 when excluding values before 1990), with the mean of ES(s) of each abstract. The temporal evolution is decreasing, with τ value of -0.88 (P < 0.001). (B) Scatter plot of the temporal evolution of yearly mean of proportion of statistically significant ES per abstract (N = 243 832 when excluding values before 1990). There is a monotonic upward trend: τ value = 0.88 (P < 0.001) The results of this cross-study conceptual replication are in accordance with results from the original studies. Of course, the validation of the results only concerns the results presented here, which have emerged only because of the presence of common variables in both databases. This conceptual replication process could not check other important results reported in the original studies. Because the size of the Effect Sizes Study’s database was around three times larger than that of the original study, it had more statistical power to detect subtle phenomena and our conceptual replication adds some evidence of a slight re-increase of error rates over time. This was only superficially mentioned in the Error Study, which showed a slight rise of hazard ratio errors between 2000 and 2016. Georgescu and Wren’s explanation concerning the recently increased use of HR, once confined to high-impact journals, remains plausible. We would also suggest two alternative explanations: (i) This trend could be attributed to random variation (the positive slope has been close to 0 for both ORs and HRs since 2005, and is not recovered for RRs). (ii) By generalizing the authors’ explanation, we can hypothesize that the proliferation of ‘mainstream journals’ could have led to a deterioration in the scientific quality of original studies. Although the number of scientific journals indexed in PubMed has grown regularly since 1965, a clear accentuation of such a phenomenon can be observed since 2005 (Bowen and Casadevall, 2015). It will be interesting to follow the evolution of this trend in the near future, for example by updating the present analysis. In conclusion, as replication is a cornerstone of scientific evidence, it is important for editors to keep room for this kind of analysis and encourage authors to share their whole database along with source codes of scripts, particularly in computational research. The authors thank Ms Susan Becker for her assistance with English language editing. This work was supported by Toulouse University Hospital (CHU de Toulouse), by Toulouse University (Université Paul Sabatier), the Midi-Pyrenees region, the research platform of the Toulouse Dental Faculty (PLTRO) and by the French National Research Agency (Agence Nationale de la Recherche—ANR—http://dx.doi.org/10.13039/501100001665) under grant ANR-16-CE18-0019-01. Conflict of Interest: none declared.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.