How to Validate Predictive Immunohistochemistry Testing in Pathology?
Bibliographic record
Abstract
To the Editor.—I read with interest a recent editorial by Erik Thunnissen entitled “How to Validate Predictive Immunohistochemistry Testing in Pathology? A Practical Approach Exploiting the Heterogeneity of Programmed Death Ligand-1 Present in Non-Small Cell Lung Cancer.”1 It would be significant and desirable if laboratories could simplify different spheres of validation of predictive immunohistochemistry (IHC) biomarkers. The introduction of programmed death ligand-1 (PD-L1) testing for immunotherapy started a new, more complex era for IHC assay development and validation. The challenges that laboratories face forced us to rethink what type of laboratory test IHC is, what “fit-for-purpose” validation means, and a few other parameters that were first embraced by drug development and pharmaceutical research rather than by anatomic pathologists.2 In following their steps, we have discovered how to distinguish analytic/technical sensitivity and specificity from diagnostic sensitivity and specificity, and that this distinction is essential in the validation of predictive biomarkers.3,4 Thunnissen explored the role of what he termed “critical samples, which have an epitope concentration close to the threshold of the validated assay,” the type of samples that were traditionally used by proficiency testing programs to assess calibration/analytic sensitivity of the IHC assays, and were also previously termed “descriptive limit of detection” and incorporated in IHC Critical Assay Performance Controls (iCAPCs) in 2015.3 These samples are derived from human tissues or other sources (cell lines, xenografts, etc) and can be used to demonstrate basic analytic sensitivity, specificity, and reproducibility as described for iCAPCs.4 However, testing of 20 positive and 20 negative samples still applies for technical validation because their purpose is not to show analytic sensitivity, but that the assay protocol performs as it should in a representative set of clinical samples (eg, specific tumor type) by demonstrating reportable range, cellular localization, tissue distribution, results with the representative range of preanalytic conditions, etc.5 Furthermore, any clinical validation, including “indirect clinical validation” that is used by the author in this editorial, requires at least demonstration of assay diagnostic accuracy in comparison with an established reference standard (or “diagnostic accuracy criteria”) and, for the purpose of comparison of methods, at least 50 positive and 50 negative samples are recommended by the CLSI DP12-A2 User Protocol for Evaluation of Qualitative Test Performance.6 The purpose of the indirect clinical validation is to ensure that the candidate test has the same or nearly the same diagnostic accuracy as the comparator assay (reference test/diagnostic accuracy criteria), and therefore that it is safe for patient selection for a specific therapy. Although analytic sensitivity and specificity are related to diagnostic sensitivity and specificity, no studies have yet demonstrated that we can make direct assumptions from one to the other. Although calibration and optimization of the assay are greatly helped with “critical samples” (aka iCAPCs), to benchmark analytic sensitivity, unfortunately these samples do not tell us too much about diagnostic accuracy and are not sufficient for indirect clinical validation. I am concerned that taking the shortcut approach for indirect clinical validation for predictive biomarkers could compromise patient safety.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.017 | 0.080 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.002 | 0.001 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.002 | 0.004 |
| Scholarly communication | 0.006 | 0.009 |
| Open science | 0.004 | 0.002 |
| Research integrity | 0.013 | 0.028 |
| Insufficient payload (model declined to judge) | 0.004 | 0.006 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".