Measuring regional impact: The case for bigger data

Global research impact can be measured using a diverse and ever-growing array of tools. Impact can be analysed with respect to the journal, the research institution, or the individual scholar from which research outputs emanate. However, we do not currently have metrics to measure the impact of research in individual geographical regions. Why would regional impact metrics be useful? Research outputs tend to concentrate in a relatively small area: the USA and Europe. In our field, emergency medicine, this area delivers nearly 60% of all research outputs recorded on Scopus despite making up less than 20% of the global population. Time and again it has been found that the findings from a high-income setting do not translate to the low-income setting: patients differ, pathogens vary, and infrastructures diverge in their human, physical, and technological resources. Quite literally, what is life-saving in Los Angeles may be lethal in Lusaka (Andrews et al., 2017). In low- and middle-income countries (LMICs), identifying the journals with greatest regional impact would enable local librarians to identify the most regionally impactful journals, local clinicians to discover context-appropriate research (to translate into practice), and local policy-makers to direct the regional research agenda accordingly. Perhaps most importantly, identifying the most influential research outputs within individual regions may act as a stimulus for regional research in LMICs and thus begin to redress the research imbalance between the West and the rest. How should one begin to construct a regional impact metric? Currently, the impact of clinical research is primarily measured using citation-based metrics. In general, these metrics are not set up to capture regional impact as there is no geographical nuance in citation calculations: a citation from Africa is given the same weight as a citation from Europe. This lack of granularity masks wide variations in the influence of research across regional contexts. A regional impact metric for Africa, for instance, should be designed so that a citation from Cameroon carries more weight than a citation from Canada. One faces a preliminary hurdle, however, when one tries to define where a citation is ‘from’. A citation is a binary relation between two entities: the original output (and its author(s)) and the citing output (and its author(s)). As a first pass, one may identify the geographical location of impact by examining the affiliation of the original output's author. Some citation databases, like Scopus, do exactly this in order to generate research rankings for different countries and world regions. However, this approach immediately leads to counterintuitive results: a paper written by US-based authors about a research project conducted in Rwanda will show no African impact according to the Scopus regional measure even if cited widely by Rwandan researchers. Hence it seems that regional impact cannot be reliably established by examining the affiliation of the original author. Maybe, then, it would be more fruitful to locate research impact using the affiliation of the citing author. According to this approach, an article cited by a Rwandan researcher would register as having impact in Rwanda; this seems intuitively correct. However, painting a map of research impact in this way would make for rather sparse cartography. Twenty-six countries in the African region have produced no academic output in emergency medicine journals in the past 5 years. It follows that no author from any of these countries has cited material in an emergency medicine journal. Thus, according to this approach (which equates impact with citations from authors affiliated to the region), no emergency medicine research has had any impact in these 26 countries at all. Moreover, citation volumes are largely linked to publication volumes, and it was already established that the latter is dominated by the USA and Europe; therefore, a regional impact metric that looks at the affiliation of the citing author would inevitably be skewed towards locating ‘impact’ in the USA and Europe. In clinical medicine, it does not follow that research has not had impact in a specific region simply because it has not been cited by researchers from that region. Not all clinicians are researchers, and vice versa; healthcare providers may read research and act upon it without citing it. This observation highlights a key challenge to the use of citations as a proxy for describing clinical research impact: clinical research can, and does, have real impact without needing to be cited. Although the citation offers a simple, straightforward measure for use in impact metric calculations, it fails to capture the impact a research output has had on clinical practice. A metric that reliably measures this is likely to remain elusive. Another straightforward measure that is routinely captured in the Scopus database along with citations is online views. Online views are less vulnerable to the objection tabled above: while clinical research need not be cited to have impact, it is difficult to imagine how clinical research could have impact without being read. Another advantage of online views is that they can be accurately located geographically. Every device connecting to a website must give an IP address in order for the website to send its information to the right place. Typically, the public IP address reveals the city or institution where a device is located. For example, if a website receives a request from an IP address that starts 137.158.x.x, it knows the request has come from the University of Cape Town. Therefore, online views can, in theory, provide far more granular data about the location of impact than citations. Finally, online views allow for more detailed cartography of impact than citations: there may not have been any emergency medicine publications from Angola in the past 5 years, but there are certainly clinicians with internet access who view research. View-based metrics can capture this activity, which citation-based metrics ignore. To explore the use of views as a metric within emergency medicine further, we replaced citations with views and recalculated standard bibliometric measures, based on view data taken from the Scopus database. First, when ranking African outputs in emergency medicine, Resuscitation – which periodically produces highly cited guidelines, drops from first to sixth place; and Annals of Emergency Medicine, typically considered a top-tier journal for impact, falls from second to outside the top 10. Second, at a paper level, it transpires that there is very little correlation between views and citations. Taking the example of the African Journal of Emergency Medicine, an open-access journal, we found the Pearson correlation coefficient between article citations and views to be 0.156 – that is, no correlation whatsoever. These two observations suggest that regional view data may provide insights both at the level of the journal, identifying publications which tend to disseminate content relevant to a particular region, and at the level of the article, highlighting those which may not have had significant research impact (few citations) but may have had clinical impact (many views). The final surprise comes from looking at the actual numbers. Often, a paper has more citations than it has views; in fact, this occurs in 10% of articles in the Scopus dataset we looked at. There are two possible reasons for this: that the Scopus view data are incomplete and that papers are being viewed in ways that are not captured by Scopus; or that researchers cite papers without reading them. Likely, both explanations are true. In particular, the Scopus dataset does not differentiate between an online view and a PDF download (both are treated as ‘full-text requests’); it has been noted that downloaded and printed documents are often circulated between researchers (Tenopir et al., 2017), and this may deflate the view statistics for articles. Indeed, using Scopus alone to measure online views may skew research metrics towards high-income settings. Institutions in the West are more likely to be able to afford the subscription fees for Elsevier titles while users without institutional access are more likely to read versions held on personal or open-access repositories. While Scopus records the former activity in its online view data, it does not (and, as things stand, cannot) record the latter. Therefore, the sketch of communication drawn from Scopus likely under-represents views from low-income settings. The observation that the Scopus view data are incomplete also provides an opportunity. Suppose for a moment that Scopus were not run by Elsevier but by Facebook or Google. How would they approach this problem? We suggest they would, as a minimum, seek to improve the usefulness of the data they capture in two domains. First, there would be more focus on determining user location. Again, looking at African Journal of Emergency Medicine, location data are currently only recorded when users access articles via an institutional login – however, this represents less than one quarter of view requests. In the remaining 75% of cases, the information about the region, country, and continent are lost and the user is simply recorded as being a ‘guest’. Not so for Google. Google tailors results based on the incoming IP address: you do not receive advertising for companies based in Newcastle when searching in Nova Scotia. Capturing the location of online view data could generate a wealth of insights into where research is being read. These insights may inter alia help researchers identify local research trends and facilitate discovery. Second, there would be greater efforts made to capture the nature of online views. A citation is an all-or-nothing phenomenon – either one paper cites another, or it does not. An online view is a more complex entity: it can last a few seconds, it can consist of a user immediately scrolling to the references section, or it can be repeated by the same user multiple times over the course of a month. Facebook Analytics routinely logs the length of time each user spends watching a video; it records the proportion of people who view a video to completion, and the proportion who stop watching within 10 seconds; it measures the number of ‘unique’ viewers of a piece of content and also the number of repeat viewers. These data are then fed back to marketers who can use these data to evaluate the impact of their message. The potential parallels to academic research are clear: were ScienceDirect to use a similar approach to capture these data it could, for instance, establish if an article had been read, skimmed, or clicked past; it could ascertain if more time had been spent looking at the abstract, conclusions, or references, and so on. In principle, an online view seems to be able to tell us far more about the quality of interaction between the reader and output than a citation. Therefore, it would appear that with improved strategies to capture data, online views would be well-suited to measuring regional impact in clinical research and other disciplines. The principle is not new – the mid-2000s saw a number of attempts to add usage data to the arsenal of metrics available to quantify scholarly communication. In the USA, one group proposed the ‘usage impact factor’ (Bollen & Van de Sompel, 2008); in the United Kingdom, another group developed the ‘usage factor’ (Shepherd, 2013). Both movements, however, were unable to promote their measures to the ranks of the Impact Factor. We maintain that their impetus should not be lost, and that online views are uniquely able to capture regional impact of research. However, there are two broad objections to this line of argument. The first is practical. A single article may be viewed by perhaps a dozen different routes, however, comparatively, there is only one way to view a Facebook page, and that is via Facebook. For this reason, online views will not be able to provide robust and comparable metrics for scholarly communication as they do for social media analytics, because it will be impossible to collate views reliably from multiple different sources. Indeed, it was noted above that Scopus online usage statistics are incomplete and likely skewed. This argument is valid if somewhat defeatist. Currently, any user who reads an open-access ScienceDirect article as a ‘Guest’ user does not have their country or region (visible from their IP address) recorded. Were ScienceDirect to log this, as Google and Facebook do, this alone would generate rich information about research impact and scholarly communication practices; this fruit is low-hanging. Moreover, there are ongoing efforts to standardize the collection of online usage data in order to create comparable statistics. In the UK, Project COUNTER has developed a Code of Practice that allows views to be compared across different publishers and online repositories (Fleming-May & Grogg, 2012). It is common practice for librarians to measure the online usage of a particular resource as part of establishing its value to the collection; however these data are collected piecemeal and often remain uncollated. Were COUNTER or a similar organization able to collect global view data from across the main publishers and larger online repositories, the implications for understanding the regional impact of research would be far-reaching. In 2016, the ‘pirate’ repository Sci-Hub released their usage data, including the IP addresses of its users (Bohannon, 2016) – this demonstrated both the technical feasibility of generating a global ‘usage’ map and the richness of the insights that such data may contain. The second objection is academic. Whereas a citation suggests – although as we saw above, does not confirm – that an article has been read, digested, and used by a researcher, a view does nothing of the sort. Someone could view an article and find it boring, useless, or incorrect; all would be treated the same by using a view-based metric. Therefore, views are not the building block needed to establish what is high-quality, impactful research. However, if a view analysis shows a person has spent 20 min scrolling through an article, then would this not seem more, not less, suggestive of engagement and impact than a citation? Of course, citation-based metrics certainly have their place, and may remain a more appropriate unit for measuring impact than views in many fields. However, creating metrics based on views certainly would expand the bibliometric options, and this is surely no bad thing. By capturing more view data, it would be possible to offer a different measure of impact, and a far more granular view of the regional distribution of research consumption. It would not require an overhaul of the major journal and article repository websites for these data to be collected and made available. Academia.edu, for example, already routinely records both incoming IP addresses and how users scroll through its articles; however, this information is mainly kept for internal use (Zach Foster, personal communication). What would a world with these data collection improvements look like? There may be some adverse side effects. Impact-hungry researchers or unscrupulous journals could more easily manipulate view figures than citations by, for example, creating web robots that would systematically read and re-read their articles. However, it has been demonstrated that citations are vulnerable to similar tactics (Delgado López-Cózar, Robinson-García, & Torres-Salinas, 2014); Altmetrics, too, can be manipulated by a range of strategies (Adie, 2013). Will the potential benefits outweigh the potential risks? We believe so. By collecting regional view data, journals may be incentivised to increase their view-footprint beyond the West; this would disproportionately benefit the developing world. As a view, unlike a citation, requires access to the full-text, this may also incentivise journals to widen access to their articles. Moreover, libraries would be empowered to establish which journals have impact in their own region, rather than having to approximate this using a global average. Local data can ensure subscriptions deliver the greatest returns. By focusing on views (and view duration) as well as citations, authors would be encouraged to increase the clarity, readability, and engagingness of their work. Reaping these rewards would not require a data revolution: the data are there, they are just not being collected. It appears then, that in spite of all the hype surrounding big data, that in the field of scholarly communication, data are not yet big enough.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.