Number needed to test: quantifying risk stratification provided by diagnostic tests and risk predictions
Bibliographic record
Abstract
Risk stratification is the ability of a test or model to separate those at high vs. low risk of disease. There is no risk stratification metric that is in terms of the number of people requiring testing, which would help with considering the benefits, harms, and costs associated with the test and interventions. We introduce the expected number needed to test (NNtest) to identify one more disease case than by randomly selecting people for disease ascertainment. We show that NNtest measures risk stratification, allowing us to decompose NNtest into components that contrast the increase in risk upon testing positive (‘concern’) versus the decrease in risk upon testing negative (‘reassurance’). A graph of the reciprocals of concern vs. reassurance have linear contours of constant NNtest, visualizing the relative importance and tradeoff of each component to better understand the properties of risk thresholds with equal NNtest. We apply NNtest to the controversy over the risk threshold for who should get testing for BRCA1/2 mutations that cause high risks of breast and ovarian cancers. We show that risk thresholds between 0.78% and 5% optimize NNtest. At these thresholds, NNtest≈120 people will require risk-model evaluation to find one more mutation-carrier. However, these thresholds of equal NNtest provide very different concern and reassurance, with 0.78% providing much more reassurance (and thus much less concern) than 5%. Given that genetic testing costs are declining rapidly, the greater reassurance provided by the 0.78% threshold might be deemed more important than the greater concern provided by the 5% threshold.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.020 | 0.162 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.003 | 0.002 |
| Science and technology studies | 0.001 | 0.002 |
| Scholarly communication | 0.004 | 0.006 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.003 | 0.003 |
| Insufficient payload (model declined to judge) | 0.006 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".