S840 Evaluation of Machine Learning Models for the Assessment of the Endoscopic Mayo Score in Ulcerative Colitis: A Systematic Review
Bibliographic record
Abstract
Introduction: The endoscopic Mayo Score (eMS) is intended to provide an objective measurement of endoscopy and is a critical component of related endpoints in Ulcerative Colitis (UC) clinical trials. Wide variability in eMS grading has been reported among central readers, contributing to inconsistency in endoscopic results. Machine learning (ML) models offer a standardized solution. Adoption of an automated eMS model requires testing to demonstrate model performance and generalizability. The objective of this study is to provide a systematic review on testing of ML eMS prediction models on full-length endoscopic video recordings from patients with UC. Methods: Studies evaluating video-level eMS prediction models on UC endoscopy video datasets (independent of data used in model training) were included. We included all full-length manuscripts from human studies published in English. PubMed/MEDLINE, EMBASE, and Web of Science were systematically searched on December 31, 2023, and supplemented by reference checks and Google search. Three rounds of title screening were conducted. Information on test set characteristics and performance on clinically relevant endpoints were extracted independently by 2 authors, with disparities resolved through discussion. Results: Five studies met criteria for inclusion, reporting data from 6 unique test cohorts. Five cohorts were internal test cohorts with 2 involving trial data (n=134-147 videos) and 3 involving data from routine care (n=27-51 videos). One cohort was an external test cohort which involved trial data (n=264 videos). Definition of the reference standard (i.e. ground truth) varied across studies with 2 cohorts reporting adjusted analyses based on modification to the definition of the reference standard. Accuracy in predicting ordinal eMS grades (0, 1, 2, 3) ranged from 56.8-83.3%. Accuracy in predicting eMS 0, 1 vs 2, 3 and eMS 0 vs 1, 2, 3 (each aligned with a definition of endoscopic improvement and remission in trials) ranged from 84-90.2% and 90-95.5%, respectively. Conclusion: Several studies have reported promising data on the performance of ML models to determine video-level eMS grades as determined by human readers in UC. This technology may ultimately provide less biased endoscopic assessments and improve standardization across clinical trials in UC. Further validation and consistency of test dataset characteristics are required to ensure model generalizability and to enable comparison across models (Figure 1).Figure 1.: Description of test dataset characteristics and model performance against key endpoints. Key endpoints include ordinal eMS, eMS 0, 1 vs 2, 3 (a definition of endoscopic improvement in trials), and eMS 0 vs 1,2,3 (a definition of endoscopic remission in trials). All test sets are independent of those used in model training. Red indicates testing on a clinical trial dataset. Blue indicates testing on a routine care dataset. * indicates testing on an external test set (relative to an internal test set from the same site or a holdout of the model training dataset). ^ indicates cohort results in an adjusted analysis based on modification to the definition of the reference standard. Acc, accuracy.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.021 | 0.074 |
| Meta-epidemiology (narrow) | 0.002 | 0.001 |
| Meta-epidemiology (broad) | 0.011 | 0.012 |
| Bibliometrics | 0.012 | 0.009 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.003 | 0.003 |
| Open science | 0.003 | 0.001 |
| Research integrity | 0.002 | 0.001 |
| Insufficient payload (model declined to judge) | 0.006 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".