Scholarly Metrics under the Microscope: From Citation Analysis to Academic Auditing
Bibliographic record
Abstract
Blaise Cronin and Cassidy R. Sugimoto., eds. Scholarly Metrics Under the Microscope: From Analysis to Academic Auditing. Series: ASIST Monograph. Medford: Information Today. 2015. 976 pp. ISBN-13: 978-1573874991What began life as a tool for retrieving the literature of science, ..., has become, unwittingly, the basis of a system for calibrating scientific performance and shaping careers, (Cronin and Sugimoto).This quote will resonate with many academics/researchers. Given the current emphasis on performance and reputation building, as demonstrated by academics' on-going quest to increase citation rates and h-index, it is easy to lose sight of the original intent of citation indexes; even more so when one discovers that some administrators in academe see a relationship between salaries and citation rates (Jonker and Hicks).This book consists of a representative cross-section of writings published on scholarly metrics between 1955 and 2014 and is meant to serve as a one-stop resource that focuses on the theoretical, conceptual, methodological, and ethical concerns associated with scholarly metrics. It is also a comprehensive and critical reader of the birth and development of scholarly metrics as reflected in the literature. The papers (55 of them) are essentially reprints of the originals as only minimal changes (such as the removal of fax numbers, e-mail addresses and such) were made.The documents are not arranged in chronological order, but the first one takes the reader back to the 1955 paper by Eugene Garfield: Citation Indexes for Science: A New Dimension in Documentation Through Association of Ideas, originally published in Science. Reading this paper reminds one that the original intent was to create a tool for literature research, to provide researchers with the ability to trace the origin of ideas and to eliminate the uncritical citation of fraudulent, incomplete or obsolete data by making it possible for the conscientious scholar to be aware of criticisms of earlier papers. (Cronin and Sugimoto).In working through this volume, the variety of readings is refreshing. The editors collected scholarly papers, editorial commentaries, blog posts, and letters that elucidate current trends and explain the history of how we arrived at the various applications of scholarly metrics.This mammoth work, comprising almost 1000 pages, is arranged according to six different themes: 1) and theories, 2) Validity issues, 3) Data sources, 4) Indicators, 5) Science policy, and 6) Systemic effects. Each section consists of between seven and eleven contributions. As the editors indicate in the introduction, one could possibly argue for a different arrangement of these papers according to different criteria, but these themes appear to work well enough to produce a framework that is conducive to exploring the breadth and depth of scholarly metrics and its many variants The editors start each of the six sections with a scene-setting introduction which is helpful in providing additional context for the papers collected in that specific section. It is also beneficial to have the goals of the section presented to the reader in these introductions.In the first section, Concepts and Theories, some attention is given to the problems arising from the way in which citations are used. It is pointed out that references to other papers are done for various reasons, and that counting and comparing the numbers of citations would be a valid exercise only if these references were normatively governed and consistently applied. However, since such behaviour is inherently subjective, depending on circumstance and authors' motivation, the use thereof for faculty evaluations is highly questionable at best. Some attention is also given to more recent developments, such as webometrics and altmetrics, pointing out that there are many questions that remain unanswered in the use of these metrics. …
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Direct model labels (unvalidated)
Per-model category and study-design labels from the labeling rounds. They are machine output, unvalidated, and the disagreement between models ships as data. No study design here is MEDLINE-validated yet.
| Model arm | Categories | Study design | Confidence |
|---|---|---|---|
| gemma | MetaresearchBibliometrics Domain: Evaluation · Genre: Empirical About the Canadian research system: no · About a Canadian topic: no | Not applicable | low |
| gpt | BibliometricsMetaresearch Domain: Evaluation · Genre: Commentary About the Canadian research system: no · About a Canadian topic: no | Not applicable | low |
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.063 | 0.106 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.034 | 0.105 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.017 | 0.042 |
| Open science | 0.001 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedLabeled directly by 2 models reading the full record.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".