Letter to the Editor: Models Developed Using Small Datasets Should be Appropriately Evaluated

The study by León-Justel et al (1) describes the development of a model for identifying individuals at increased risk of Cushing’s syndrome. Unfortunately, as we will highlight, a number of methodological shortcomings cast doubt on the usefulness of the model. Our first point relates to sample size. The effective sample size for prediction model studies is not the number of individuals in the dataset, but rather the number of individuals experiencing the event of interest; in this case, only 26 individuals developed Cushing’s syndrome. Exacerbating the situation further is the large number of variables examined to be predictors of Cushing’s syndrome. To develop a prediction model, the rule of thumb is that a minimum of 10 events-per-variable (EPV) are required to reduce the risk of overfitting (2), and much higher values are often needed (3). The current study examined at least 23 predictors, yielding an EPV of 26/23 = 1, considerably lower than the value of 10. When the number of events is rare (in relation to the number of predictors examined), alternative approaches, based on penalization, have been shown to provide better predictions (4). Regardless of the approach, particularly in instances of low EPV, it is crucial to carry out a fair evaluation of the predictive accuracy of the model. Bootstrapping is widely recommended as the preferred approach for internal validation (5). León-Justel et al (1) carried out bootstrapping, but unfortunately, it appears that this was done incorrectly. It is important that all variable selection procedures are replayed in each bootstrap sample (including the inappropriate univariate screening as carried in the León-Justel study). Bootstrapping the final model, ie, evaluating the final model in each bootstrap sample, will produce a biased estimate of the model performance. As such, we believe the estimates of model discrimination (ie, area under the receiver operating characteristic curve) are optimistically too high. As well as assessing discrimination, it is recommended that model calibration also be assessed, as indicated in the TRIPOD Statement for reporting prediction model studies (6). In the study of León-Justel et al (1), the authors assessed calibration by calculating the Hosmer-Lemeshow test. This test, while common, has been shown to be a poor assessment of calibration. It assesses neither the direction nor the magnitude of any (mis)calibration and is highly influenced by sample size, often showing favorable results in small sample sizes (7). Calibration should ideally be assessed graphically by plotting predicted outcome probabilities (x-axis) against observed outcomes (y-axis) using a high-resolution smoothed (loess) line with confidence limits (8). The direction and magnitude of any miscalibration can then be examined across the entire probability range. We recommend that the authors and other investigators developing prediction models consult the TRIPOD Statement (www.tripod-statement.org) for key information to report when describing its development and validation (6) so that readers have the minimal information required to judge the quality of the study. The TRIPOD Explanation and Elaboration paper (5) highlights the rationale of the importance of transparent reporting but also discusses various methodological considerations that investigators should consider when developing and validating a prediction model. Disclosure Summary: The authors report no conflicts of interest. events-per-variable.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.