External validity: the neglected dimension in evidence ranking
Bibliographic record
Abstract
Evidence that is both accurate (internally valid) and relevant (externally valid) is needed to decide which treatment is best for a particular patient. Evidence rankings facilitate the marshalling of evidence on clinical decisions in the common context of an overwhelming number of studies, some with conflicting results. Evidence from randomized control trials is typically ranked above evidence from non-experimental studies since rankings are based primarily, if not exclusively, on considerations of internal validity. We propose that evidence rankings should consider equally both internal and external validity. External validity includes how closely the study population, the institution types in the study, the types of physicians in the study, the role of clinician decision-making (e.g. dose adjustment) in the study, and the role of patient preferences in the study resemble those in actual practice. The example of spironolactone use in heart failure illustrates the danger in using evidence that is internally but not externally valid. Ideally, a treatment should only be used when both internally and externally valid evidence indicates that it will be useful for the particular patient.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.018 | 0.029 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.001 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".