Altmetrics versus traditional bibliometrics

Editor, Traditional bibliometrics have been criticised for manipulations like retractions1 and self-citations.2 Altmetrics based on social media, news, policy documents and patents are increasingly becoming popular, with the Altmetric.com the older provider using weighted count sources, each one contributing in a different magnitude to the altmetric attention score (AAS) of the output.3,4 According to the most recent revision the sources contribution to the AAS are: News (8), Blog (5), Policy document (3), Patent (3), Wikipedia (3), Twitter (1), Peer review(1), F1000 (1), Syllabi (1), Facebook (0.25), Reddit (0.25), Q&A (0.25), Youtube (0.25), Number of Mendeley readers (0), Number of Dimentions and Web of Science (WoS) citations (0).4 In the current study, we hypothesised that citation counts will continue to increase overtime in contrast to AAS and we investigate the changes of the AAS and of citation counts in five anaesthesia journals with a WoSs (Clarivate Analytics’ since 2017) Impact Factor. Articles published in five anaesthesia journals during the year 2016 were included in the study. The five anaesthesia journals representing both Europe and North America are Anesthesia and Analgesia (A&A), Anesthesiology, British Journal of Anaesthesia (BJA), Canadian Journal of Anesthesia (CJA) and European Journal of Anaesthesiology (EJA). In a previous study, the AAS of articles with a nonzero AAS were searched and recorded from October 2016 up to July 2017.4 We also conducted a search of the citation counts that these articles received up to November 2017. In the current study, a search was further conducted from 20 April 2018 until 3 June 2018 for the cumulative AAS and in June 2018 for the citation counts of these same articles. The dataset collected included all articles (original articles, reviews, editorials, practice guidelines and correspondence publications) with nonzero AAS, obtained by searching the articles online (tracked from Altmetric.com) and citations counts of these articles obtained from Thomson's WoS (Clarivate Analytics’ since 2017) database. Publications with zero AAS, book reviews, infographics and errata were excluded from the analysis. Outcomes of the study were the change in the overall AAS and in the overall citation counts between the two periods of measurements and the differences in the AAS and in the citations counts obtained by each individual journal between 2017 (previous study) and 2018. Changes in AAS between the years 2017 and 2018 for each journal, and for the journals overall, were compared using the Wilcoxon paired samples signed rank test. The citation counts were compared using the paired samples t test. Of the 2015 articles identified, 145 did not fulfil the criteria of the study protocol, and 573 had a zero AAS. Results were analysed for the remaining 1297 articles4 (Fig. 1). BJA could not be assessed for AAS change as the new publisher (Elsevier) who took over in 2018 suspended the Altmetric.com and implemented different alternative metrics known as plumX metrics. This different metrics provider is based on different variables, therefore does not allow comparison between the two time measurements and evaluation of the AAS changes by time.Fig. 1: Flow diagram of the study.Absolute numbers, medians and upper and lower quartiles of cumulative AAS for the years 2017 and 2018 are shown in Table 1. Statistically significant differences in the AAS between the years 2017 and 2018 were observed for the journals A&A, Anesthesiology and CJA. The overall AAS changes for the time period of the study though statistically significant are about 5% and not practically substantial.Table 1: Median values and lower upper quartiles of the Altmetric Attention Scores and mean and SD values of the citation counts for the years 2017 and 2018All journals exhibited a significant increase in the citation counts in the year 2018 compared with the year 2017 (P < 0.01 for each journal comparison) (Table 1). From November 2017 to June 2018 we found an increase in citations in the A&A by 56%, in the Anesthesiology journal by 58%, in BJA by 69%, in the CJA by 55% and in the EJA by 36%. The overall AAS were also increased but changes though statistically significant, were marginal (about 5%). The overall increase in the citation counts is 58%, a tenfold increase compared with the AAS change. Thus, almost a year later the citations pattern is characterised by a different course compared with the AAS. AAS are associated with citation counts. Thelwall et al.5 reported association between high citation counts and high AAS. The AAS of cardiovascular outputs published in the eight WoS Journals with the highest impact factor correlates with their three year obtained citation counts.6 The AAS for six PLoS journals were found to correlate with the citations retrieved from the WoS with likelihood for the high Web Citation counts to correlate with the high AAS values.7 Different disciplines may have an impact on these correlations.7 These findings are consistent with the results of our previous study where AAS of articles published in 2016 in the five anaesthesia journals correlated with their citation counts.4 Nevertheless, this correlation is not related to patterns of growth these metrics follow over a relatively short period of time as we observed an increase in the numbers of citations ten times higher compared with the increases found in the AAS. The AAS may predict the citation numbers of an article. However, correlation does not necessarily reflect a causal effect. The two variables have different life courses over time as citations’ increase is long-lasting leaving the AAS behind. In contrast to citations, the AAS does not always relate to expertise on the topic. It may or may not predict the scholar future of a research outcome in contrast to high citation numbers over time, which are associated with the endurance of the article to the test of time. Limitations of the present study are that only journals of the same discipline were considered, articles with zero altmetrics were excluded, and comparisons of AAS for the BJA were impossible as since 2018, the journal's metrics is the PlumX, which counts less tweets, fewer news medium, fewer blogs and higher numbers of Mendeley readers.8 In conclusion, citation counts increase in significantly higher rate than the AAS over time. These results imply that AAS, a metric partly outside the scientific world, highlights the impact an output may have in science in real time; citations remain the solid indicator of the scientific impact of an output over time. Our findings cannot be generalised and further studies are required to refute or confirm them.

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.