Geopolitical Forecasting Skill in Strategic Intelligence
Bibliographic record
Abstract
Abstract Extending research by the authors on intelligence forecasting, the forecasting skill of 3622 geopolitical forecasts extracted from strategic intelligence reports was examined. The codable subset of forecasts (N = 2013) was expressed with verbal probabilities (e.g., likely) and translated to numeric probability equivalents. This subset showed very good calibration and discrimination, but also underconfidence. There was no support for the hypothesis that forecasting skill was good mainly because of the general ease of forecasting topics. First, forecasting skill was as good among authoritative key judgments as in the general set. Second, forecasts that were assigned high degrees of certainty, indicative of ease, (p ≤ 0.05 or p ≥ 0.95) did not discriminate as well as less certain forecasts (0.05 < p < 0.95), and these subsets did not differ in calibration. Sensitivity and benchmarking tests further revealed that if the 1609 uncodable forecasts were all assigned forecast probabilities of .5 (i.e., if all followed a “cautious ignorance” rule), skill characteristics would still show a large effect size improvement over a variety of guesswork strategies. The findings support a cautiously optimistic assessment of forecasting skill in strategic intelligence and indicate that such skill is not primarily attributable to the selection of easy forecasting topics. However, the large proportion of uncodable cases suggests that intelligence forecasts could be improved by avoiding imprecise language that affects not only the codability but also, in all likelihood, the interpretability and indicative value of forecasts for intelligence consumers. Copyright © 2017 John Wiley & Sons, Ltd.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.006 | 0.070 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.000 | 0.001 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.003 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".