2772 AI assisted reader evaluation in CT head interpretation (AI-REACT): results from a multi-case multi-reader study
Bibliographic record
Abstract
Aims and Objectives A non-contrast CT head scan (NCCTH) is the most common cross-sectional imaging investigation requested in the Emergency Department (ED), and several Artificial Intelligence (AI) tools have been developed to detect abnormalities on NCCTH, however there is currently little real-world evidence to support adoption. The AI-REACT study (IRAS 310995, NCT05427838) evaluated the impact of AI algorithm on the diagnostic performance of ED clinicians, radiologists and radiographers. Method and Design A retrospective dataset of 150 NCCTH was compiled, including 63 normal control cases and 97 abnormal cases containing intracranial haemorrhage (ICH), infarct, midline shift, mass effect, or skull fracture. 30 readers of varying experience were recruited across four NHS trusts including 10 general radiologists, 15 Emergency Medicine clinicians, and five CT radiographers. Readers interpreted each scan first without, then with, the assistance of the qER EU 2.0 AI tool, with an intervening 2-week washout period. Using an arbitrated consensus opinion of 2 neuroradiologists as ground truth, the stand-alone performance of qER was assessed, and its impact on the readers’ diagnostic performance analysed. Results and Conclusion Pooled analyses demonstrated a significant increase in reader sensitivity for abnormal scans (0.828 to 0.897, +0.069, 95%CI +0.106 to +0.0136, p >0.001) and ICH (0.846 to 0.916, +0.07 95%CI 0.108 to 0.0321 p = >0.001). ED clinicians with AI assistance demonstrated a sensitivity of 0.879 (abnormality) and 0.948 (ICH) compared to unaided radiologist sensitivity 0.890 (abnormality) and 0.939 (ICH), with no statistically significant changes in specificity. Use of AI-assisted image interpretation led to a significant increase in the ability of ED clinicians to accurately identify abnormality and ICH on CT Head scans, to a level comparable to that of radiologists. These important findings should be fully explored prospectively. Further analysis of the effects on pathology and reader subgroups will help to identify potential strengths and use cases for this application.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.059 | 0.143 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.004 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.003 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".