Semi-automated Literature Review for Scientific Assessment of Socioeconomic Climate Change Scenarios
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Climate change is now recognized as a global threat, and the literature surrounding it continues to increase exponentially. Expert bodies such as the Intergovernmental Panel on Climate Change (IPCC) are tasked with periodically assessing the literature to extract policy-relevant scientific conclusions that might guide policymakers. However, concerns have been raised that climate change research may be too voluminous for traditional literature review to adequately cover. It has been suggested that practices for literature review for scientific assessment be updated/augmented with semi-automated approaches from bibliometrics or scientometrics. In this study, we explored the feasibility of such recommendations for the scientific assessment of literature around socioeconomic climate change scenarios, so-called Shared Socioeconomic Pathways (SSPs). For automated literature reviews, most methods can be subsumed under two broad categories of classification tasks that use either (1) Natural Language Processing (NLP) or (2) Citation Networks. We performed two levels of classification tasks: (1) identifying SSP articles from a large corpus of climate change research and developing a database of SSP-related articles; (2) classifying SSP articles into different sectoral categories. We applied three machine learning algorithms for the text classification task: Multinomial Naïve Bayes, Logistic Regression, and Linear Support Vector Classification. However, the vocabulary of the SSP literature too closely resembles the vocabulary of broader climate change research for an NLP approach to be effective. We then attempted a citation network approach. We compared two sets of different community detection algorithms (the Louvain algorithm and the Fluid community detection algorithm), with one iteration of each algorithm containing 8 clusters and the next set containing 16. The citation network approach outperformed NLP with respect to false negatives. It also provided the ability to assess the uptake of SSPs across different sectors of climate change research. We concluded that, at the time of the study, the SSP corpus may not yet be large enough or diverse enough from broader climate change research for applying machine learning techniques for automated literature review. However, our research suggests that until there is a critical mass of SSP studies, there is the potential to divide labor between human and machine readers. Some of the data collection tasks currently done by human author teams, such as assessing scenario research, could be semi-automated to ensure and enhance the coverage of the literature. We also drew conclusions about the uptake of the SSP framework over its first 5 years in the broader climate change research literature. We observed that the uptake of SSPs in certain sub-disciplines (e.g., food systems) progressed slowly. Hence, to keep SSPs relevant, it may be fruitful to target SSP studies to particular research communities (e.g., sectors with slower uptake).
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.002 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.001 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it