Diagnostic performance and agreement of auditors for evaluation of computer-aided optical polyp diagnosis: Prospective study
Bibliographic record
Abstract
Background and study aims: Guidelines recommend independent auditing of diagnostic performance for clinical implementation of computer-aided optical polyp diagnosis (CADx). This study evaluated diagnostic performance and interobserver agreement of auditors and offered guidance on conducting CADx audits. Methods: Images and videos of all ≤ 5-mm polyps from a large, prospective study with systematic activation of CADx were audited by three expert endoscopists. Experts performed independent, blinded diagnostic review including documentation of confidence level. The primary outcome was sensitivity of audit by three experts for high-confidence adenomas compared with pathology. Secondary outcomes included number of reviewers for optimal CADx auditing and interobserver agreement. Results: Four hundred eighty-seven diminutive polyps were audited (510 patients). Sensitivity was 99.4% (95% confidence interval [CI] 96.0-100) using three experts (Strategy A); 88.7% (95% CI 84.1-92.1) using two experts and one referee (Strategy B); 99% (95% CI 96-99.8), 98.8% (95% CI 95.4-99.8), and 99.4% (95% CI 96.3-100) using two-expert combinations (Strategy C); and 98.2% (95% CI 95.1-99.4), 97.3% (95% CI 94.0-98.9), and 88.9% (95% CI 83.6-92.7) for each expert individually (Strategy D). Among 266 pathology-based adenomas, Strategy A evaluated 160 polyps versus 196, 172, and 170 in Strategy C; and 220, 223, and 207 in Strategy D. Strategy B evaluated all 266 adenomas. Overall interobserver agreement was moderate (kappa 0.52), but very high for high-confidence adenomas (kappa 0.89). Conclusions: Expert audit for evaluating CADx resulted in high sensitivity and interobserver agreement for high-confidence adenomas. Audit by two experts, with a third expert for arbitration, permitted audit of all polyps and effective assessment of CADx within clinical studies.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.060 | 0.150 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.003 | 0.002 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".