Evaluating the Bank Of England Density Forecasts
Bibliographic record
Abstract
We consider evaluating the UK Monetary Policy Committee’s inflation density forecasts using probability integral transform goodness-of-fit tests. These tests evaluate the whole forecast density. We also consider whether the probabilities assigned to inflation being in certain ranges are well calibrated, where the ranges are chosen to be those of particular relevance to the MPC, given its remit of maintaining inflation rates in a band around 2 12 % per annum. Finally, we discuss the decision-based approach to forecast evaluation in relation to the MPC forecasts. Every quarter since August 1997 the Bank of England Inflation Report has pub-lished density forecasts of the annual rate of retail price inflation (excluding mortgage interest repayments, the RPIX measure) made by the Monetary Policy Committee. This marks an important departure from the traditional concern with the central tendency or most likely outcome of the future value of the variable, and is in line with the increasing recognition that an assessment of the degree of uncertainty surrounding a point forecast is generally indispensable. The particular form of the forecast densities emphasises possible asymmetries between upside and downside risks, presumably because these are foremost in the minds of the Monetary Policy Committee (MPC) during their deliberations over the base rate. The Bank of England’s own assessment of these forecasts has tended to focus on the closeness of some measure of the central tendency and the outcomes,1 although Wallis (2003) shows how the Christoffersen (1998) likelihood ratio goodness-of-fit statistics for evaluating interval forecasts can be interpreted as Pearson ‘chi-squared goodness-of-fit ’ statistics, and evaluates the Bank density forecasts within this framework. This paper pursues a number of approaches to the evaluation of the MPC’s forecasts. First, they are evaluated as forecast densities using goodness-of-fit tests based on the probability integral transform. Tests based on the probability integral transform have recently been used to evaluate probability distributions of macro-economic variables, such as the distributions of expected inflation from the Survey
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.027 | 0.169 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.003 | 0.003 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.004 | 0.004 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.002 | 0.001 |
| Insufficient payload (model declined to judge) | 0.007 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".