Abstract TP51: When Can Aspects be Read Reliably?
Bibliographic record
Abstract
Introduction: Alberta Stroke Program Early CT Score (ASPECTS) is a systematic approach to assess early ischemic change on non-contrast CT (NCCT). Concerns have however been expressed about its reliability when making clinical decisions in patients with acute ischemic stroke. We chose to systematically assess technical, environmental and patient specific variables that potentially affect ASPECTS interpretation. Methods: We randomly selected 150 patients with acute ischemic stroke from the PRoveIT database. All patients had baseline NCCT and CT angiography head and neck. Three raters (expert, fellow and trainee) read ASPECTS on the same NCCT three times (Sessions 1-3) at minimum interval of 10-14 days. Raters were kept blinded to follow-up data throughout the study. No baseline clinical information was provided in Session 1. Raters were provided clinical information (age, baseline NIHSS and side of stroke) in session 2 and additional multiphase CTA in session 3. Reading environment [room light and time pressure (<60 s for interpretation) vs. core lab] was altered during readings. Data on motion artifact, leukoaraiosis, old infarcts on NCCT were collected. Time taken for ASPECTS interpretation was collected across all the readings. Reliability was assessed using Intra-cluster correlation coefficient (ICC). Results: The highest inter-rater reliability was found in session 3 (ICC 0.47; p<0.001). The rest of the analyses was restricted to session 3. Reliability in session 3 was not affected by time pressure or ambient light settings (all p<0.01). In session 3, patient motion (ICC 0.35 present vs. 0.49 absent) and old infarcts (ICC 0.42 present vs. 0.48 absent) worsened reliability; however presence of leukoaraiosis did not affect reliability (ICC 0.48 present vs. 0.46 absent). Mean time for ASPECT interpretation by trainee, fellow and expert were 38.9 s (+/-12.8s), 49.8 s (+/-15.4s) and 38.9 s (+/-14s) respectively. Conclusion: ASPECTS interpretation on NCCT is most reliable when clinical and CTA information is available. Interpretation with this information is reliable even in a well-lit room and under time pressure, the environment that mimics real life acute stroke.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.022 | 0.151 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.001 | 0.001 |
| Science and technology studies | 0.001 | 0.001 |
| Scholarly communication | 0.003 | 0.003 |
| Open science | 0.001 | 0.001 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.006 | 0.005 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".