Artificial Intelligence-assisted reader evaluation in acute CT head interpretation (AI-REACT): a multireader multicase study
Bibliographic record
Abstract
Abstract Background Non-contrast CT head scans (NCCTH) are the most frequently requested cross-sectional imaging in the Emergency Department. While AI tools have been developed to detect NCCTH abnormalities, most validation studies compare AI to radiologists, with limited evidence on the impact of AI assistance for other healthcare professionals. Objective To evaluate whether an AI-powered tool improves the accuracy, speed, and confidence of general radiologists, emergency clinicians, and radiographers in detecting critical abnormalities on NCCTH, and to assess the tool’s stand-alone performance and factors influencing diagnostic accuracy and efficiency. Methods A retrospective dataset of 150 NCCTH (52 normal, 98 with critical abnormalities: intracranial haemorrhage, hypodensity, midline shift, mass effect, or skull fracture) was reviewed by 30 readers (10 radiologists, 15 emergency clinicians, 5 radiographers) from four NHS trusts. Each reader interpreted scans first unaided, then with the qER EU 2.0 AI tool, separated by a 2-week washout. Ground truth was established by consensus of two neuroradiologists. We assessed the stand-alone performance of qER and its effect on reader diagnostic accuracy, confidence, and interpretation speed. Results The qER algorithm demonstrated strong diagnostic performance across most pathology subgroups (AUC 0.821–0.976). With AI assistance, pooled reader sensitivity for critically abnormal scans increased from 82.8% to 89.7% (+6.9%, 95% CI +1.4% to +10.6%, p<0.001), and for intracranial haemorrhage from 84.6% to 91.6% (+7.0%, 95% CI +3.2% to +10.8%, p<0.001), but specificity decreased from 84.5% to 78.9% (–5.5%, 95% CI –11.0% to –0.09%, p=0.046). Reader confidence AUC did not change significantly. ED clinicians with AI achieved sensitivity comparable to unaided radiologists, with no significant change in specificity. Conclusion AI-assisted interpretation increased reader sensitivity for critical abnormalities but reduced specificity. Notably, AI assistance enabled ED clinicians to reach diagnostic sensitivity similar to unaided radiologists, supporting the potential for AI to extend the diagnostic capabilities of non-radiologists. Further prospective studies are warranted to confirm these findings in real-world settings. Funding This study was funded by Qure.ai via an NHSX Award Ethics The study has been approved by the UK Healthcare Research Authority (IRAS 310995, approved 13/12/2022). The use of anonymised retrospective NCCTH has been authorised by Oxford University Hospitals. Trial registration number NCT06018545 . Research in context What is already known on this topic AI-derived algorithms for the detection of pathological findings on non-contrast CT head (NCCTH) images have previously demonstrated strong diagnostic performance when used on retrospective datasets. AI-assisted image interpretation using these algorithms has been shown to enhance the diagnostic performance of general and neuro-radiologists in silico . The potential for AI to enhance the performance of less skilled readers who may encounter and be required to act on these images in clinical practice (e.g. non-specialist radiologists, emergency medicine clinicians and radiographers) is as yet untested, however. What this study adds This large multicase multireader study demonstrates that AI-assisted image interpretation may be used to enhance the in silico diagnostic performance of Emergency Department physicians to a level comparable to that of general radiologists. How this study might affect research, practice or policy This study raises the possibility that AI-assisted image interpretation could be used to assist non-radiologist clinicians in the safe interpretation of NCCTH scans. Further prospective research is required to test this hypothesis in clinical practice and explore the potential for AI-assisted interpretation to support safe discharge of patients with normal or low-risk scans.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.041 | 0.130 |
| Meta-epidemiology (narrow) | 0.001 | 0.001 |
| Meta-epidemiology (broad) | 0.001 | 0.003 |
| Bibliometrics | 0.003 | 0.003 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.002 | 0.002 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".