97 Statistical and systematic uncertainties affect biomarker reliability: a case study in multiplex immunofluorescence
Bibliographic record
Abstract
Background Every measurement comes with statistical uncertainties, arising from the finite data sample, and systematic uncertainties, from errors in the measurement or model. Biomedical analyses typically account only for the statistical uncertainty from the number of patients. These are reported as p-values, power calculations, and other similar metrics.As datasets grow and this statistical uncertainty shrinks, it is not necessarily the dominant uncertainty anymore. A closer look at other uncertainties that affect analyses is critically needed.1 Methods We processed mIF slides from four cohorts containing a total of n=75 pre-treatment specimens from patients with NSCLC using the AstroPath platform, 2 which applies numerous image corrections and produces comprehensive cell maps.We identified CD8+FoxP3+ cells as a positive predictor of response to anti-PD1 immunotherapy in non-small-cell lung cancer (NSCLC). We further developed the Diversity of Niches Unlocking Treatment Sensitivity (DONUTS) biomarker. The DONUTS are niches that mimic the neighborhoods around CD8+FoxP3+ cells but are 150x more common than the cells themselves.3 4 We evaluated both biomarkers’ performance and calculated the statistical uncertainties resulting both from the finite number of patients and from the finite number of cells or DONUTS within each patient‘s biopsy.We processed our data through the AstroPath pipeline again, removing some of the corrections. We estimated the systematic uncertainties that would have been present had we not applied the corrections and the remaining systematic uncertainties resulting from imperfect corrections.We developed the ROC Picker package5 to propagate these uncertainties to ROC and Kaplan-Meier curves.Results With the relatively small cohort sizes (largest: n=25) used in this analysis, the statistical uncertainty from the number of patients still dominates, but other uncertainties contribute appreciably.The statistical uncertainty from the number of CD8+FoxP3+ cells is comparable to the statistical uncertainty from the number of patients (figure 1A), but the statistical uncertainty from the more common DONUTS is much smaller (figure 1B).The systematic uncertainties were also substantial before the corrections in the pipeline, but dropped by more than half after applying those corrections.Conclusions We demonstrate a proof of concept for evaluating statistical and systematic uncertainties in mIF. While standard practice is to consider only one statistical uncertainty, other uncertainties are becoming more important as datasets grow and, if not accounted for, will impact biomarkers and clinical decision making. 6 Our methodology can be applied to biomarkers from all data modalities, ensuring that they remain reliable in the era of big data.References Berry S, Giraldo N, Green B, et al. Analysis of multispectral imaging with the AstroPath platform informs efficacy of PD-1 blockade. Science. 2021;372.Green B, et al. AstroPath Pipeline v0.1.0. Zenodo. 2021.Cottrell T, Roskes J, Cohen E, et al. Early, effector CD8+FoxP3+ cells and their topology associate with outcomes in patients with non-small cell lung carcinoma (NSCLC) receiving neoadjuvant anti-PD-1-based therapy. In preparation. 2025.Cohen E, et al. CD8+FoxP3+ cells represent early, effector T-cells and predict outcomes in patients with resectable non-small cell lung carcinoma (NSCLC) receiving neoadjuvant anti-PD-1-based therapy. J Immunother Cancer. 2022;10:A63.Roskes J. ROC Picker v1.1.0. GitHub. 2024.Taube J, Sunshine J, et al. Society for immunotherapy of cancer: updates and best practices for multiplex immunohistochemistry (IHC) and immunofluorescence (IF) image analysis and data sharing. J Immunother Cancer. 2025;13.Abstract 97 Figure 1Kaplan-Meier curves for regression-free survival for patients with NSCLC, stratified by density of (A) CD8+FoxP3+ cells or (B) DONUTs. The error bands show the statistical error resulting from the finite number of cells or DONUTs
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.070 | 0.255 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.002 | 0.003 |
| Scholarly communication | 0.003 | 0.002 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.003 | 0.002 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".