Letter to the Editor: Models Developed Using Small Datasets Should be Appropriately Evaluated
Bibliographic record
Abstract
The study by León-Justel et al (1) describes the development of a model for identifying individuals at increased risk of Cushing’s syndrome. Unfortunately, as we will highlight, a number of methodological shortcomings cast doubt on the usefulness of the model. Our first point relates to sample size. The effective sample size for prediction model studies is not the number of individuals in the dataset, but rather the number of individuals experiencing the event of interest; in this case, only 26 individuals developed Cushing’s syndrome. Exacerbating the situation further is the large number of variables examined to be predictors of Cushing’s syndrome. To develop a prediction model, the rule of thumb is that a minimum of 10 events-per-variable (EPV) are required to reduce the risk of overfitting (2), and much higher values are often needed (3). The current study examined at least 23 predictors, yielding an EPV of 26/23 = 1, considerably lower than the value of 10. When the number of events is rare (in relation to the number of predictors examined), alternative approaches, based on penalization, have been shown to provide better predictions (4). Regardless of the approach, particularly in instances of low EPV, it is crucial to carry out a fair evaluation of the predictive accuracy of the model. Bootstrapping is widely recommended as the preferred approach for internal validation (5). León-Justel et al (1) carried out bootstrapping, but unfortunately, it appears that this was done incorrectly. It is important that all variable selection procedures are replayed in each bootstrap sample (including the inappropriate univariate screening as carried in the León-Justel study). Bootstrapping the final model, ie, evaluating the final model in each bootstrap sample, will produce a biased estimate of the model performance. As such, we believe the estimates of model discrimination (ie, area under the receiver operating characteristic curve) are optimistically too high. As well as assessing discrimination, it is recommended that model calibration also be assessed, as indicated in the TRIPOD Statement for reporting prediction model studies (6). In the study of León-Justel et al (1), the authors assessed calibration by calculating the Hosmer-Lemeshow test. This test, while common, has been shown to be a poor assessment of calibration. It assesses neither the direction nor the magnitude of any (mis)calibration and is highly influenced by sample size, often showing favorable results in small sample sizes (7). Calibration should ideally be assessed graphically by plotting predicted outcome probabilities (x-axis) against observed outcomes (y-axis) using a high-resolution smoothed (loess) line with confidence limits (8). The direction and magnitude of any miscalibration can then be examined across the entire probability range. We recommend that the authors and other investigators developing prediction models consult the TRIPOD Statement (www.tripod-statement.org) for key information to report when describing its development and validation (6) so that readers have the minimal information required to judge the quality of the study. The TRIPOD Explanation and Elaboration paper (5) highlights the rationale of the importance of transparent reporting but also discusses various methodological considerations that investigators should consider when developing and validating a prediction model. Disclosure Summary: The authors report no conflicts of interest. events-per-variable.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.038 | 0.361 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.003 | 0.002 |
| Bibliometrics | 0.001 | 0.002 |
| Science and technology studies | 0.002 | 0.002 |
| Scholarly communication | 0.004 | 0.005 |
| Open science | 0.004 | 0.002 |
| Research integrity | 0.023 | 0.027 |
| Insufficient payload (model declined to judge) | 0.008 | 0.006 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".