Concordance with SPIRIT-AI guidelines in reporting of randomized controlled trial protocols investigating artificial intelligence in oncology: a systematic review
Bibliographic record
Abstract
BACKGROUND: Artificial intelligence (AI) is a promising tool used in oncology that may be able to facilitate diagnosis, treatment planning, and patient management. Transparency and completeness of protocols of randomized controlled trials (RCT) involving AI interventions is necessary to ensure reproducibility of AI tools across diverse clinical settings. The SPIRIT 2013 and SPIRIT-AI 2020 guidelines were developed as evidence-based recommendations for complete reporting of trial protocols. However, the concordance of AI RCT protocols in oncology to SPIRIT reporting guidelines remains unknown. This systematic review evaluates the concordance of protocols of RCTs evaluating AI interventions in oncology to the SPIRIT 2013 and SPIRIT-AI 2020 reporting guidelines. METHODS: A systematic search of Ovid Medline and Embase was conducted on October 22, 2024 for primary, peer-reviewed RCT protocols involving AI interventions in oncology. Eligible studies were screened in duplicate and data extraction assessed concordance to SPIRIT 2013 and SPIRIT-AI 2020 guideline items. Item-specific concordance was measured as the proportion of studies that reported the item. Average concordance was measured as the median proportion of items reported for each study. RESULTS: Twelve RCT protocols met the inclusion criteria. The median concordance to SPIRIT 2013 guidelines was 81.92% (IQR 74.88-88.95) and the median concordance to SPIRIT-AI 2020 guidelines was 78.21% (IQR 67.21-89.20). For SPIRIT 2013 guidelines, high concordance was observed for items related to study objectives and ethics, but gaps were identified in reporting blinding procedures, participant retention, and post-trial care. For SPIRIT-AI 2020 guidelines, there remained gaps based on data quality management, performance error analysis, and accessibility of AI intervention code. CONCLUSION: While concordance to reporting guidelines in oncology AI RCT protocols was moderately high, critical gaps in protocol reporting persist that may hinder reproducibility and clinical implementation. Future efforts should focus on increasing awareness and reinforcement to enhance reporting quality necessary to foster the responsible integration of AI into oncology practice.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.667 | 0.855 |
| Meta-epidemiology (narrow) | 0.003 | 0.006 |
| Meta-epidemiology (broad) | 0.019 | 0.025 |
| Bibliometrics | 0.022 | 0.026 |
| Science and technology studies | 0.005 | 0.011 |
| Scholarly communication | 0.014 | 0.016 |
| Open science | 0.010 | 0.010 |
| Research integrity | 0.011 | 0.008 |
| Insufficient payload (model declined to judge) | 0.007 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; the direct Gemma label and the distilled Codex classifier agree on what is shown here.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".