PD27-10 TEMPORAL VALIDATION OF AN ARTIFICIAL INTELLIGENCE TOOL (SEPERA) TO INFORM NERVE-SPARING STRATEGY DURING RADICAL PROSTATECTOMY AND COMPARISON AGAINST UROLOGISTS
Bibliographic record
Abstract
You have accessJournal of UrologySurgical Technology & Simulation: Artificial Intelligence II (PD27)1 May 2024PD27-10 TEMPORAL VALIDATION OF AN ARTIFICIAL INTELLIGENCE TOOL (SEPERA) TO INFORM NERVE-SPARING STRATEGY DURING RADICAL PROSTATECTOMY AND COMPARISON AGAINST UROLOGISTS Lauren Pickel, Kevin Zhang, Ryan Booth, Aiman Shahid, Maximiliano Ringa, Amna Ali, Amy Chan, Nathan Perlis, Robert J. Hamilton, Neil E. Fleshner, Antonio Finelli, Alistair E. W. Johnson, Girish S. Kulkarni, Andrew Feifer, Alexandre R. Zlotta, and Jethro C. C. Kwong Lauren PickelLauren Pickel , Kevin ZhangKevin Zhang , Ryan BoothRyan Booth , Aiman ShahidAiman Shahid , Maximiliano RingaMaximiliano Ringa , Amna AliAmna Ali , Amy ChanAmy Chan , Nathan PerlisNathan Perlis , Robert J. HamiltonRobert J. Hamilton , Neil E. FleshnerNeil E. Fleshner , Antonio FinelliAntonio Finelli , Alistair E. W. JohnsonAlistair E. W. Johnson , Girish S. KulkarniGirish S. Kulkarni , Andrew FeiferAndrew Feifer , Alexandre R. ZlottaAlexandre R. Zlotta , and Jethro C. C. KwongJethro C. C. Kwong View All Author Informationhttps://doi.org/10.1097/01.JU.0001008580.58088.27.10AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Accurate prediction of side-specific extra-prostatic extension (ssEPE) is crucial to inform nerve-sparing (NS) strategy. We recently developed an artificial intelligence (AI) model, SEPERA (Side-specific Extra-Prostatic Extension Risk Assessment), to predict risk of ssEPE and validated it on an international, multi-institutional cohort of patients from 2008-2020. As ongoing validation of AI models is key, we aimed to further assess SEPERA and compare its recommendations against urologists. METHODS: Temporal validation was performed on 695 patients undergoing radical prostatectomy at two academic and community hospitals in Canada from 2020-2022. Patients that received prior radiation or hormone therapy were excluded. Primary outcome was presence of ssEPE on prostatectomy. SEPERA was compared to a logistic regression (LR) model from the original study, which included identical variables as SEPERA. Models were assessed based on discrimination, calibration, and decision curve analysis at decision thresholds between 0-30%, as commonly used in other studies. NS strategy at the time of surgery was compared to SEPERA's recommendations (complete, partial, or minimal NS). Proportion of cases with ssEPE and positive surgical margins (ssPSM) was examined. RESULTS: ssEPE was found in 467 out of 1390 prostatic lobes (34%). SEPERA performed similar to the original study with AUROC 0.75 (95% CI 0.73-0.78) and AUPRC 0.63 (95% CI 0.58-0.67). SEPERA outperformed LR, which achieved AUROC 0.73 (95% CI 0.70-0.75, p=0.002) and AUPRC 0.61 (95% CI 0.56-0.65, p=0.02). SEPERA was well-calibrated for risks between 15-45% and had higher net benefit compared to LR for clinically relevant risk thresholds between 15-25%. SEPERA's recommendations differed from clinical decisions in 49% of cases (Figure. 1). Where SEPERA recommended "Minimal" NS but more was performed, 52% of cases had ssEPE and 35% had ssPSM. Where SEPERA recommended "Complete" NS but less was performed, only 13% of cases had ssEPE and 3% had ssPSM. CONCLUSIONS: We further show the generalizability of SEPERA in both academic and community settings. SEPERA has the potential to improve pathological outcomes, particularly if decisions are guided by SEPERA in cases where "Minimal" NS is recommended. Download PPT Source of Funding: This project was supported by the Temerty Centre for AI Research and Education in Medicine Summer Research Studentship © 2024 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 211Issue 5SMay 2024Page: e554 Advertisement Copyright & Permissions© 2024 by American Urological Association Education and Research, Inc.Metrics Author Information Lauren Pickel More articles by this author Kevin Zhang More articles by this author Ryan Booth More articles by this author Aiman Shahid More articles by this author Maximiliano Ringa More articles by this author Amna Ali More articles by this author Amy Chan More articles by this author Nathan Perlis More articles by this author Robert J. Hamilton More articles by this author Neil E. Fleshner More articles by this author Antonio Finelli More articles by this author Alistair E. W. Johnson More articles by this author Girish S. Kulkarni More articles by this author Andrew Feifer More articles by this author Alexandre R. Zlotta More articles by this author Jethro C. C. Kwong More articles by this author Expand All Advertisement PDF downloadLoading ...
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.014 | 0.066 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.001 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.002 | 0.001 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.009 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".