Fantastic Models and How to Find Them: A Literature Review on Model Selection Practices
Bibliographic record
Abstract
When starting a new modelling task, one of the very first decisions the modeller has to make is the choice of which model(s) to use. That this selection can have a significant impact on our model results has been shown through numerous studies (e.g. Melsen et al. [2018], Mendoza et al. [2015]). That it is often based on legacy (habit, practicality, convenience) rather than adequacy (fitness for purpose) has also been recognized (Addor&Melsen [2019]). We present the results of a literature review on previous model selection practices to better understand what modellers have considered important when choosing a model for a particular purpose.We analyze more than 250 studies discussing model selection, model intercomparison or multi-model studies with a focus on conceptual hydrologic models. We identify the criteria used to determine which models were considered “fit for purpose” and why. The analyzed studies compare between two and 7488 model structures in two to 1013 basins. We aggregate information on the evaluation criteria used for different modelling purposes and in different locations and identify common model selection strategies. We monitor the range of model performance in individual comparisons and document both, the progress made and the challenges faced during previous model comparisons.Our analysis shows a strong dependency on aggregated statistical metrics and a tendency for simplified calibration approaches that were meant to support a broad range of evaluation practices. This often led to a lack of clear answers on which models to prefer. The reasons that were given for (not) selecting a specific model structure seem to indicate a mismatch between the perceptions of when model adequacy is reached. We therefore conclude with a critical discussion of previous model selection strategies and call for a more nuanced approach to model evaluation as well as standards for reporting modelling practices and results.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.028 | 0.097 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.003 | 0.004 |
| Bibliometrics | 0.010 | 0.010 |
| Science and technology studies | 0.001 | 0.003 |
| Scholarly communication | 0.007 | 0.010 |
| Open science | 0.007 | 0.004 |
| Research integrity | 0.002 | 0.003 |
| Insufficient payload (model declined to judge) | 0.008 | 0.003 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".