Concordance with CONSORT-AI guidelines in reporting of randomised controlled trials investigating artificial intelligence in oncology: a systematic review
Bibliographic record
Abstract
Background: The advent of artificial intelligence (AI) tools in oncology to support clinical decision-making, reduce physician workload and automate workflow inefficiencies yields both great promise and caution. To generate high-quality evidence on the safety and efficacy of AI interventions, randomised controlled trials (RCTs) remain the gold standard. However, the completeness and quality of reporting among AI trials in oncology remains unknown. Objective: This systematic review investigates the reporting concordance of RCTs for AI interventions in oncology using the CONSORT (Consolidated Standards of Reporting Trials) 2010 and CONSORT-AI 2020 extension guideline and comprehensively summarises the state of AI RCTs in oncology. Methods and analysis: We queried OVID MEDLINE and Embase on 22 October 2024 using AI, cancer and RCT search terms. Studies were included if they reported on an AI intervention in an RCT including participants with cancer. Results: This study included 57 RCTs of AI interventions in oncology that were primarily focused on screening (54%) or diagnosis (19%) and intended for clinician use (88%). Among all 57 RCTs, median concordance with CONSORT 2010 and CONSORT-AI 2020 was 82%. Compared with trials published before the release of CONSORT-AI (n=8), trials published after the release of CONSORT-AI (n=49) had lower median overall CONSORT (82% vs 92%) and CONSORT 2010 (81% vs 92%) concordance but similar CONSORT-AI median concordance (93% vs 93%). Guideline items related to study methodology necessary for reproducibility using the AI intervention, such as input data inclusion and exclusion, algorithm version, low quality data handling, assessment of performance error and data accessibility, were consistently under-reported. When stratifying included trials by their overall risk of bias, trials at serious risk of bias (57%) were less concordant to CONSORT guidelines compared with trials at moderate (71%) or low (84%) risk of bias. Conclusion: Although the majority of CONSORT and CONSORT-AI items were well-reported, critical gaps related to reporting of methodology, reproducibility and harms persist. Addressing these gaps through consideration of trial design to mitigate risks of bias coupled with standardised reporting is one step towards responsible adoption of AI to improve patient outcomes in oncology.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.054 | 0.443 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.030 | 0.001 |
| Bibliometrics | 0.001 | 0.002 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; both teacher heads agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".