Quantifying random variability in decision-making in pediatric cardiac surgery
Bibliographic record
Abstract
Objective: Random variability in day-to-day decision-making referred to as "noise" is associated with variation that negatively affects both the reproducibility and quality of decision-making. Although well described in other fields, the prevalence and significance of noise in medical decision-making are understudied and largely unknown. The goal of this study was to quantify noise in medical decision-making using a noise audit. Methods: A noise audit was completed by 71 (n = 71) Heart Centre staff at the Hospital for Sick Children and Seattle Children's Hospital and involved a series of cases and questions surrounding decisions commonly encountered in the care for patients with transposition of the great arteries and critical aortic stenosis. Entropy was used to quantify total variation in responses. Because absolute entropy was not immediately comparable across audit questions with a different number of prespecified options, we reported a standardized version of entropy, which was calculated by dividing the absolute entropy by its theoretical maximum for each case. To compare responses in easy to hard questions across years of experience (<10 years and >10 years) and role, we reported aggregate entropy ratios. Aggregate entropy ratios were calculated by first stratifying by group of comparison and then calculating the standardized entropy for each question and taking the average of standardized entropies for easy questions and hard questions. Finally, to determine the ratio (easy to hard), we divided the average standardized entropy of the easy questions by the average standardized entropy of the hard questions. Results: The overall audit aggregate entropy ratio was 0.85 less than 1, indicating lower-complexity questions had less variation than higher-complexity questions. The aggregate entropy ratio for those with more than 10 years of experience was 0.8 and 0.87, respectively, for those with less than 10 years of experience. The aggregate entropy ratios for those in cardiac critical care, cardiology, and cardiovascular surgery were 0.85, 0.83, and 0.96, respectively. Conclusions: Noise was pervasive in medical decision-making in the analysis of our responses to common medical decisions made for patients with congenital heart disease and can be quantified in a manner that facilitates comparisons.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.020 | 0.136 |
| Meta-epidemiology (narrow) | 0.000 | 0.000 |
| Meta-epidemiology (broad) | 0.000 | 0.000 |
| Bibliometrics | 0.002 | 0.001 |
| Science and technology studies | 0.000 | 0.001 |
| Scholarly communication | 0.001 | 0.001 |
| Open science | 0.000 | 0.002 |
| Research integrity | 0.000 | 0.001 |
| Insufficient payload (model declined to judge) | 0.001 | 0.000 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".