Going beyond journal classification for evaluation of research outputs
Bibliographic record
Abstract
Purpose Seeks to characterise world astronomy research during the last decade by an analysis of papers in the Science Citation Index identified with a special filter and to study Indian output in order to identify the leading institutions and authors. Design/methodology/approach Lists of specialist journals and title words of papers were selected to create a filter giving high precision and recall for astronomy papers. Some biology papers were erroneously retrieved because of ambiguous title words. Potential citation impact was determined from journal citation scores, and multiple regression was used to evaluate leading countries. Findings Title words added almost a quarter to the list of papers in specialist journals, and the final file contained over 96,000 papers. Potential impact increased with more authors per paper and more addresses; it was greater for papers from Canada, the UK and the USA, and less for papers from China, India and Russia; for other countries the effects of the author's location on potential impact were not statistically significant. Indian astronomy output has increased in potential impact, partly through greater international co‐authorship, but also through indigenous papers. Research limitations/implications The study was confined to one subject area, and impact was determined on the basis of journals, not of individual papers. Practical implications Use of title words in addition to journal lists is essential to sub‐field definition in order to have high precision and recall. Because of the confounding effects of authorship numbers, it is necessary to use multiple regression analysis in order to see whether research from a given country is significantly better or worse than average. Originality/value Characterises world astronomy research during the last decade by an analysis of papers in the Science Citation Index identified with a special filter.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.111 | 0.427 |
| Meta-epidemiology (narrow) | 0.003 | 0.001 |
| Meta-epidemiology (broad) | 0.005 | 0.006 |
| Bibliometrics | 0.059 | 0.068 |
| Science and technology studies | 0.003 | 0.004 |
| Scholarly communication | 0.020 | 0.018 |
| Open science | 0.004 | 0.007 |
| Research integrity | 0.002 | 0.003 |
| Insufficient payload (model declined to judge) | 0.015 | 0.006 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".