Association Between Peer Reviewers' Priority Ratings of Impact of Research Manuscripts With Citations and Altmetric Scores of Subsequently Published Articles in the Journal of Medical Internet Research
Bibliographic record
Abstract
Objective Peer-reviewed journals ask reviewers to rate the perceived impact or priority of a manuscript. Previous research has suggested an association between reviewer priority scores and citations.1 Altmetrics (alternative metrics) provide an alternative view on social impact (ie, uptake on factor, >5). This journal asks peer reviewers to rate the priority (defined as potential impact) of a manuscript on an ordinal rating scale with possible scores of 1, 2, 5, and 10 (highest priority). Manuscripts are typically reviewed by 2 reviewers. The mean priority score of all reviewers for a manuscript in the first review round constitutes the Manuscript Average Priority Score (MAPS). For this analysis, manuscripts were categorized into 4 quartiles (Qs), with the groups labeled as Q4 (MAPS score, ≤3) to Q1 (MAPS score, >5). The dependent variables, citations, and altmetric scores were obtained from the Dimensions database in February 2022; manuscripts and published articles were similarly stratified into quartiles, with the citation (or altmetrics) quartile Q1 containing the group of articles with the highest citation count (or altmetric score). The association between independent variables (MAPS scores) and citation or altmetric scores was measured using χ² tests for 4 × 4 contingency tables for the quartiles and using Spearman rank correlation between MAPS score ranks and citation or altmetric rank, respectively. Results The MAPS scores for 451 published articles ranged from 1.5 to 10; citations, from 0 to 253; and altmetric scores, from 1 to 849. Although both mean and median citations as well as altmetric scores were higher in the higher MAPS quartiles (Table 46), the results of χ² tests were not statistically significant for citations (P = .46) but were statistically significant for altmetric scores (P = .03). The Spearman rank correlation between citation ranks and MAPS score ranks was statistically significant but weak (ρ = .0955; r2 = .009; P = .03). In contrast, altmetric score ranks had a stronger correlation with MAPS score ranks (ρ = .1313; r2 = .017; P = .002). https://assets.underline.io/uploads/markdown_image/1/image/e9506e3de590eff1a8fbd4b75fe758f1.png Conclusions This longitudinal bibliometric cohort study found that in the Journal of Medical Internet Research, a journal whose subject matter lends itself to the type of attention measured by altmetrics, altmetric scores seemed to be better correlated than citations with a manuscript’s potential impact as assessed by reviewers. Peer reviewers may interpret priority and impact in terms of social impact, rather than citations, raising further questions about the appropriateness of citation-based metrics to measure impact as understood by reviewers. References Opthof T, Coronel R, Janse MJ. The significance of the peer review process against the background of bias: priority ratings of reviewers and editors and the prediction of citation, the role of geographical bias. Cardiovasc Res. 2002;56(3):339-346. doi:10.1016/S0008-6363(02)00712-5 Eysenbach G. Can tweets predict citations? metrics of social impact based on twitter and correlation with traditional metrics of scientific impact. J Med Internet Res. 2011;13(4):e123. doi:10.2196/jmir.2012 Araujo AC, Vanin AA, Nascimento DP, et al. What are the variables associated with altmetric scores? Syst Rev. 2021;10:193. doi:10.1186/s13643-021-01735-0 Conflict of Interest Disclosures Gunther Eysenbach reported equity in JMIR Publications. https://assets.underline.io/uploads/markdown_image/1/image/78cc867faeefd28c67dc78d1ef488c9f.png
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.034 | 0.316 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.011 | 0.011 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.004 | 0.003 |
| Open science | 0.001 | 0.003 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.006 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".