11 Visual quality control of assessment of AI-assisted high-volume CMR segmentation in the UK Biobank
Bibliographic record
Abstract
<h3>Background</h3> Automated algorithms are being used regularly to analyse cardiac magnetic resonance (CMR) images. Validating data output reliability from these methods is necessary to enable widespread adoption. We outline a visual quality control (QC) process for image analysis performed using automated batch processing methods. We aim to report the performance of automated methods and the reliability of replacing visual checks with a statistical outlier removal approach in UK Biobank CMR scans. <h3>Methods</h3> CMR scans included (n=1987) were from the UK Biobank COVID imaging study. Automated batch processing software developed by Circle Cardiovascular Imaging Inc (CVI 42) was used to extract chamber volumetric data, strain, native T1 and aortic flow data. The video outputs of the automated image analysis (~ 62,000 videos and 2000 images) were visually reviewed and rated by six experienced clinicians using a custom-built R Shiny app. The standardised approach (consisting of grading 1,2,3 for good, satisfactory or poor quality respectively) was agreed during two rounds of scoring followed by open discussion. Interobserver variability was assessed using Gwet’s second order agreement co-efficient (AC2) analysis. The data output from scans passing visual QC was compared with data from a statistical outlier removal QC method, using t-test analysis, in a subset of healthy individuals from baseline imaging (n = 1069). <h3>Results</h3> The quality of the automated image analysis was very high with >95% of scans passing the visual QC (scored 1 or 2) for all modalities of image analysis. There was good inter-observer agreement with overall AC2 of 0.91(± 0.14, 95% confidence interval (0.84,0.94)). There was no difference in the overall distribution of data and derived average values from visual QC process or statistical outlier removal in a subset of healthy individuals from this study. <h3>Conclusion</h3> The quality of automated image analysis is very high using the prototypes developed by CVI42 for the UK Biobank imaging study CMR scans. Therefore, larger UK Biobank datasets analysed using these automated algorithms do not need in-depth visual QC. Statistical outlier removal is a sufficient QC measure, with operator discretion for visual checks based on their respective population or research aim.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame distilled prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. Learned from the 10,348 direct Codex labels and 10,348 direct Gemma labels. Candidate is the union of thresholded teacher heads; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels or direct frontier model labels.
Codex and Gemma teacher scores by category
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.001 | 0.000 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.000 | 0.000 |
| Science and technology studies | 0.000 | 0.000 |
| Scholarly communication | 0.000 | 0.000 |
| Open science | 0.000 | 0.000 |
| Research integrity | 0.000 | 0.000 |
| Insufficient payload (model declined to judge) | 0.000 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one teacher head, not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".