11 Visual quality control of assessment of AI-assisted high-volume CMR segmentation in the UK Biobank
Bibliographic record
Abstract
Background Automated algorithms are being used regularly to analyse cardiac magnetic resonance (CMR) images. Validating data output reliability from these methods is necessary to enable widespread adoption. We outline a visual quality control (QC) process for image analysis performed using automated batch processing methods. We aim to report the performance of automated methods and the reliability of replacing visual checks with a statistical outlier removal approach in UK Biobank CMR scans. Methods CMR scans included (n=1987) were from the UK Biobank COVID imaging study. Automated batch processing software developed by Circle Cardiovascular Imaging Inc (CVI 42) was used to extract chamber volumetric data, strain, native T1 and aortic flow data. The video outputs of the automated image analysis (~ 62,000 videos and 2000 images) were visually reviewed and rated by six experienced clinicians using a custom-built R Shiny app. The standardised approach (consisting of grading 1,2,3 for good, satisfactory or poor quality respectively) was agreed during two rounds of scoring followed by open discussion. Interobserver variability was assessed using Gwet’s second order agreement co-efficient (AC2) analysis. The data output from scans passing visual QC was compared with data from a statistical outlier removal QC method, using t-test analysis, in a subset of healthy individuals from baseline imaging (n = 1069). Results The quality of the automated image analysis was very high with >95% of scans passing the visual QC (scored 1 or 2) for all modalities of image analysis. There was good inter-observer agreement with overall AC2 of 0.91(± 0.14, 95% confidence interval (0.84,0.94)). There was no difference in the overall distribution of data and derived average values from visual QC process or statistical outlier removal in a subset of healthy individuals from this study. Conclusion The quality of automated image analysis is very high using the prototypes developed by CVI42 for the UK Biobank imaging study CMR scans. Therefore, larger UK Biobank datasets analysed using these automated algorithms do not need in-depth visual QC. Statistical outlier removal is a sufficient QC measure, with operator discretion for visual checks based on their respective population or research aim.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.023 | 0.064 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.002 | 0.001 |
| Open science | 0.002 | 0.001 |
| Research integrity | 0.001 | 0.000 |
| Insufficient payload (model declined to judge) | 0.004 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".