S840 Evaluation of Machine Learning Models for the Assessment of the Endoscopic Mayo Score in Ulcerative Colitis: A Systematic Review

Introduction: The endoscopic Mayo Score (eMS) is intended to provide an objective measurement of endoscopy and is a critical component of related endpoints in Ulcerative Colitis (UC) clinical trials. Wide variability in eMS grading has been reported among central readers, contributing to inconsistency in endoscopic results. Machine learning (ML) models offer a standardized solution. Adoption of an automated eMS model requires testing to demonstrate model performance and generalizability. The objective of this study is to provide a systematic review on testing of ML eMS prediction models on full-length endoscopic video recordings from patients with UC. Methods: Studies evaluating video-level eMS prediction models on UC endoscopy video datasets (independent of data used in model training) were included. We included all full-length manuscripts from human studies published in English. PubMed/MEDLINE, EMBASE, and Web of Science were systematically searched on December 31, 2023, and supplemented by reference checks and Google search. Three rounds of title screening were conducted. Information on test set characteristics and performance on clinically relevant endpoints were extracted independently by 2 authors, with disparities resolved through discussion. Results: Five studies met criteria for inclusion, reporting data from 6 unique test cohorts. Five cohorts were internal test cohorts with 2 involving trial data (n=134-147 videos) and 3 involving data from routine care (n=27-51 videos). One cohort was an external test cohort which involved trial data (n=264 videos). Definition of the reference standard (i.e. ground truth) varied across studies with 2 cohorts reporting adjusted analyses based on modification to the definition of the reference standard. Accuracy in predicting ordinal eMS grades (0, 1, 2, 3) ranged from 56.8-83.3%. Accuracy in predicting eMS 0, 1 vs 2, 3 and eMS 0 vs 1, 2, 3 (each aligned with a definition of endoscopic improvement and remission in trials) ranged from 84-90.2% and 90-95.5%, respectively. Conclusion: Several studies have reported promising data on the performance of ML models to determine video-level eMS grades as determined by human readers in UC. This technology may ultimately provide less biased endoscopic assessments and improve standardization across clinical trials in UC. Further validation and consistency of test dataset characteristics are required to ensure model generalizability and to enable comparison across models (Figure 1).Figure 1.: Description of test dataset characteristics and model performance against key endpoints. Key endpoints include ordinal eMS, eMS 0, 1 vs 2, 3 (a definition of endoscopic improvement in trials), and eMS 0 vs 1,2,3 (a definition of endoscopic remission in trials). All test sets are independent of those used in model training. Red indicates testing on a clinical trial dataset. Blue indicates testing on a routine care dataset. * indicates testing on an external test set (relative to an internal test set from the same site or a holdout of the model training dataset). ^ indicates cohort results in an adjusted analysis based on modification to the definition of the reference standard. Acc, accuracy.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.