Development and external validation of prognostic models for COVID-19 to support risk stratification in secondary care
Bibliographic record
Abstract
Abstract Objectives Existing UK prognostic models for patients admitted to hospital with COVID-19 are limited by reliance on comorbidities, which are under-recorded in secondary care, and lack of imaging data among the candidate predictors. Our aims were to develop and externally validate novel prognostic models for adverse outcomes (death, intensive therapy unit (ITU) admission) in UK secondary care; and externally validate the existing 4C score. Design Candidate predictors included demographic variables, symptoms, physiological measures, imaging, laboratory tests. Final models used logistic regression with stepwise selection. Setting Model development was performed in data from University Hospitals Birmingham (UHB). External validation was performed in the CovidCollab dataset. Participants Patients with COVID-19 admitted to UHB January-August 2020 were included. Main outcome measures Death and ITU admission within 28 days of admission. Results 1040 patients with COVID-19 were included in the derivation cohort; 288 (28%) died and 183 (18%) were admitted to ITU within 28 days of admission. Area under the receiver operating curve (AUROC) for mortality was 0.791 (95%CI 0.761-0.822) in UHB and 0.767 (95%CI 0.754-0.780) in CovidCollab; AUROC for ITU admission was 0.906 (95%CI 0.883-0.929) in UHB and 0.811 (95%CI 0.795-0.828) in CovidCollab. Models showed good calibration. Addition of comorbidities to candidate predictors did not improve model performance. AUROC for the 4C score in the UHB dataset was 0.754 (95%CI 0.721-0.786). Conclusions The novel prognostic models showed good discrimination and calibration in derivation and external validation datasets, and outperformed the existing 4C score. The models can be integrated into electronic medical records systems to calculate each individual patient’s probability of death or ITU admission at the time of hospital admission. Implementation of the models and clinical utility should be evaluated. Article Summary Strengths and limitations of this study We developed novel prognostic models predicting mortality and ITU admission within 28 days of admission for patients hospitalised with COVID-19, using a large routinely collected dataset gathered at admission with a wide range of possible predictors (demographic variables, symptoms, physiological measures, imaging, laboratory test results). These novel models showed good discrimination and calibration in both derivation and external validation cohorts, and outperformed the existing ISARIC model and 4C score in the derivation dataset. We found that addition of comorbidities to the set of candidate predictors included in model derivation did not improve model performance. If integrated into hospital electronic medical records systems, the model algorithms will provide a predicted probability of mortality or ITU admission for each patient based on their individual data at, or close to, the time of admission, which will support clinicians’ decision making with regard to appropriate patient care pathways and triage. This information might also assist clinicians in explaining complex prognostic assessments and decisions to patients and their relatives. A limitation of the study was that in the external validation cohort we were unable to examine all of the predictors included in the original full UHB model due to only a reduced set of candidate predictors being available in CovidCollab. Nevertheless, the reduced model performed well and the results suggest it may be applicable in a wide range of datasets where only a reduced set of predictor variables is available. Furthermore, it was not possible to carry out stratified analysis by ethnicity as the UHB dataset contained too few patients in most of the strata, and no ethnicity data was available in the CovidCollab dataset.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.037 | 0.085 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.003 |
| Bibliometrics | 0.003 | 0.002 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.003 | 0.001 |
| Open science | 0.002 | 0.004 |
| Research integrity | 0.001 | 0.002 |
| Insufficient payload (model declined to judge) | 0.004 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".