Diagnostic performance and agreement of auditors for evaluation of computer-aided optical polyp diagnosis: Prospective study
Why this work is in the frame
A frame that forgets how it found something cannot be audited. These are the routes that admitted this work.
Bibliographic record
Abstract
Background and study aims: Guidelines recommend independent auditing of diagnostic performance for clinical implementation of computer-aided optical polyp diagnosis (CADx). This study evaluated diagnostic performance and interobserver agreement of auditors and offered guidance on conducting CADx audits. Methods: Images and videos of all ≤ 5-mm polyps from a large, prospective study with systematic activation of CADx were audited by three expert endoscopists. Experts performed independent, blinded diagnostic review including documentation of confidence level. The primary outcome was sensitivity of audit by three experts for high-confidence adenomas compared with pathology. Secondary outcomes included number of reviewers for optimal CADx auditing and interobserver agreement. Results: Four hundred eighty-seven diminutive polyps were audited (510 patients). Sensitivity was 99.4% (95% confidence interval [CI] 96.0-100) using three experts (Strategy A); 88.7% (95% CI 84.1-92.1) using two experts and one referee (Strategy B); 99% (95% CI 96-99.8), 98.8% (95% CI 95.4-99.8), and 99.4% (95% CI 96.3-100) using two-expert combinations (Strategy C); and 98.2% (95% CI 95.1-99.4), 97.3% (95% CI 94.0-98.9), and 88.9% (95% CI 83.6-92.7) for each expert individually (Strategy D). Among 266 pathology-based adenomas, Strategy A evaluated 160 polyps versus 196, 172, and 170 in Strategy C; and 220, 223, and 207 in Strategy D. Strategy B evaluated all 266 adenomas. Overall interobserver agreement was moderate (kappa 0.52), but very high for high-confidence adenomas (kappa 0.89). Conclusions: Expert audit for evaluating CADx resulted in high sensitivity and interobserver agreement for high-confidence adenomas. Audit by two experts, with a third expert for arbitration, permitted audit of all polyps and effective assessment of CADx within clinical studies.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.001 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from it