Knowingly repeating an incorrect and inefficient analysis is flawed logic

In their study, Garcia-Valentin et al. [ 1 ] evaluated the performance of EuroSCORE and EuroSCORE II in a cohort of 4034 patients undergoing cardiac surgery in Spain. While we applaud the authors for conducting this independent external validation, which is uncommon, we wish to highlight some methodological and reporting concerns so that future investigators do not replicate them. The two key components characterizing the performance of a prediction model are discrimination and calibration. Discrimination is the ability of the prediction model to differentiate between those who do and do not experience the outcome event (quantified by c -index which is equivalent to the area under the receiver operating characteristic curve). Calibration is the agreement between outcome predictions from the model and the observed outcomes. In the study by Garcia-Valentin et al. , calibration was evaluated using the Hosmer–Lemeshow test, which the authors correctly highlight as being problematic. The Hosmer–Lemeshow test has limited power to evaluate calibration, affected by sample size and grouping and gives no indication on the direction and magnitude of (mis)calibration [ 2 ]. Unfortunately, the authors disappointingly proceeded to use this test to judge calibration on the premise that this was used in the original model development study [ 3 ]. Repeating an incorrect and uninformative analysis on the grounds that it was done in the original study is flawed logic and if methodological concerns are raised, then alternative and correct analyses should be carried out. Knowingly repeating a flawed analysis even if concerns are raised, incorrectly supports and justifies its use by other investigators and the cycle is never broken. The study by Garcia-Valentin et al. raised concerns of the approach but only in the ‘Discussion’ section, which is often not read as closely as the ‘Methods’ or ‘Results’ section of a manuscript. In accordance with recent recommendations on the reporting of prediction model studies [ 2 , 4 ], calibration should be assessed graphically by plotting predicted outcome probabilities ( x -axis) against observed outcomes ( y -axis) using a high-resolution smoothed (loess) line. The direction and magnitude of any miscalibration can then be examined across the entire probability range. The calibration plot can also be supplemented with a numerical quantification of calibration by examining the calibration slope and intercept and the 0.9 quantile of the absolute prediction error [ 5 , 6 ]. We recommend investigators, peer reviewers and editors to read the recent guidance from the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) Initiative ( www.tripod-statement.org ). The TRIPOD reporting guideline for clinical prediction models, discusses key issues in the development and validation of a prediction model [ 4 ]. The TRIPOD guideline is similar to other well-known reporting guidelines (e.g. CONSORT, STROBE and PRISMA) designed to help authors, peer reviewers and journal editors in ensuring that the essential items describing the development or validation of a clinical prediction model are clearly reported. Accompanying the reporting guideline is an extensive Explanation and Elaboration article describing the rationale for the checklist item but also highlighting many methodological considerations when developing or validating a clinical prediction model [ 2 ].

Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.