Clinical Performance Evaluation of an Artificial Intelligence–Based Tool for Predicting the Presence of Obstructive Coronary Artery Disease: Protocol for a Cohort Observational Study
Bibliographic record
Abstract
Background A significant number of individuals undergoing coronary computed tomography angiography (CCTA) for suspected (CAD) have nonobstructive or no CAD. There is a need for clinically proven models that can predict the pretest probability of stable CAD and help to identify low-risk individuals. Optimizing patient stratification is of paramount importance to improve diagnostic yield and cost-effectiveness. Objective We aimed to determine whether each patient needs to undergo CCTA because of suspected CAD. The main objective of this study is to evaluate the clinical performance of an artificial intelligence (AI)-based tool in predicting significant coronary artery stenosis (>50%), as well as its utility by medical professionals. Methods Data for this study have been acquired from 750 participants as part of routine clinical practice in AHEPA (American Hellenic Educational Progressive Association) General Hospital of Thessaloniki. The dataset has several features, including demographics (eg, age, gender), medical history (eg, diabetes mellitus, arterial hypertension), and clinical variables (eg, creatinine, epicardial fat volume). At least 2 expert cardiologists and 2 expert radiologists are involved in this study, who provide the ground truth. A trained AI-based model embedded in an easy-to-use and user-friendly web application is implemented in practice. Several AI algorithms are being examined, and the model found to perform best so far is the Optimized Voting model, which is a combination of the best performing iterations of random forest and extreme gradient boosting. The performance metrics that are being used are accuracy, precision, recall, F1-score, area under the receiver operating characteristic curve, and area under the precision-recall curve. Results Recruitment for this study began in July 2023. Data collection, development, training, and deployment of the AI web tool were completed by May 2024. In total, data from 500 individuals were collected for training and internal validation, while the best performing model was validated externally in another 250 individuals. For training and internal validation, the dataset was split into 70% for training and 20% for validation and 10% for testing. Currently, the best performing model achieves an accuracy of approximately 82% in successfully predicting stenosis greater than 50%. Additionally, an explainable AI algorithm is used to provide explanations in relation to the decisions made aiming to increase the trust of the clinicians in the tool. Conclusions The proposed study represents a novel approach of a web-based AI-driven solution with explainability features for optimizing patient stratification with the goal of improving diagnostic yield and cost-effectiveness of CCTA utilization within the context of cardiology clinical practice. International Registered Report Identifier (IRRID) DERR1-10.2196/67697
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.045 | 0.040 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.004 | 0.005 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.002 | 0.002 |
| Scholarly communication | 0.002 | 0.001 |
| Open science | 0.002 | 0.002 |
| Research integrity | 0.002 | 0.003 |
| Insufficient payload (model declined to judge) | 0.010 | 0.003 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".