Verification of an Ensemble Prediction System against Observations
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Abstract A verification system has been developed for the ensemble prediction system (EPS) at the Canadian Meteorological Centre (CMC). This provides objective criteria for comparing two EPSs, necessary when deciding whether or not to implement a new or revised EPS. The proposed verification methodology is based on the continuous ranked probability score (CRPS), which provides an evaluation of the global skill of an EPS. Its reliability/resolution partition, proposed by Hersbach, is used to measure the two main attributes of a probabilistic system. Also, the characteristics of the reliability are obtained from the two first moments of the reduced centered random variable (RCRV), which define the bias and the dispersion of an EPS. Resampling bootstrap techniques have been applied to these scores. Confidence intervals are thus defined, expressing the uncertainty due to the finiteness of the number of realizations used to compute the scores. All verifications are performed against observations to provide more independent validations and to avoid any local systematic bias of an analysis. A revised EPS, which has been tested at the CMC in a parallel run during the autumn of 2005, is described in this paper. This EPS has been compared with the previously operational one with the verification system presented above. To illustrate the verification methodology, results are shown for the temperature at 850 hPa. The confidence intervals are computed by taking into account the spatial correlation of the data and the temporal autocorrelation of the forecast error. The revised EPS performs significantly better for all the forecast ranges, except for the resolution component of the CRPS where the improvement is no longer significant from day 7. The significant improvement of the reliability is mainly due to a better dispersion of the ensemble. Finally, the verification system correctly indicates that variations are not significant when two theoretically similar EPSs are compared.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it