External Validation of Scoring Instruments for Evaluating Pediatric Resuscitation
Bibliographic record
Abstract
INTRODUCTION: Although many methods have been proposed to assess clinical performance during resuscitation, robust and generalizable metrics are still lacking. Further research is necessary to develop validated clinical performance assessment tools and show an improvement in outcomes after training. We aimed to establish evidence for validity of a previously published scoring instrument--the Clinical Performance Tool (CPT)--designed to evaluate clinical performance during simulated pediatric resuscitations. METHODS: This was a prospective experimental trial performed in the simulation laboratory of a pediatric tertiary care facility, with a pretest/posttest design that assessed residents before and after pediatric advanced life support (PALS) certification. Thirteen postgraduate year 1 (PGY1) and 11 PGY3 pediatric residents completed 5 simulated pediatric resuscitation scenarios each during 2 consecutive sessions; between the 2 sessions, they completed a full PALS certification course. All sessions were video recorded. Sessions were scored by raters using the CPT; total scores were expressed as a percentage of maximum points possible for each scenario. Validity evidence was established and interpreted according to Messick's framework. Evidence regarding relations to other variables was assessed by calculating differences in scores between pre-PALS and post-PALS certification and PGY1 and PGY3 using a repeated-measures analysis of variance test. Internal structure evidence was established by assessing interrater reliability using intraclass correlation coefficients (ICCs) for each scenario, a G-study, and a variance component analysis of individual measurement facets (scenarios, raters, and occasions) and associated interactions. RESULTS: Overall scores for the entire study cohort improved by 10% after PALS training. Scores improved by 9.9% (95% confidence interval [CI], 4.5-15.4) for the pulseless nonshockable arrest (ICC, 0.85; 95% CI, 0.74-0.92), 14.6% (95% CI, 6.7-22.4) for the pulseless shockable arrest (ICC, 0.98; 95% CI, 0.96-0.99), 4.1% (95% CI, -4.5 to 12.8) for the dysrhythmias (ICC, 0.92; 95% CI, 0.87-0.96), 18.4% (95% CI, 9.7-27.1) for the respiratory scenario (ICC, 0.97; 95% CI, 0.95-0.98), and 5.3% (95% CI, -1.4 to 2.0) for the shock scenarios (ICC, 0.94; 95% CI, 0.90-0.97). There were no differences between PGY1 and PGY3 scores before or after the PALS course. Reliability of the instrument was acceptable as demonstrated by a mean ICC of 0.95 (95% CI, 0.94-0.96). The G-study coefficient was 0.94. Most variance could be attributed to the subject (57%). Interactions between subject and scenario and subject and occasion were 9.9% and 1.4%, respectively, and variance attributable to rater was minimal (0%). CONCLUSIONS: Pediatric residents improved scores on CPT after completion of a PALS course. Clinical Performance Tool scores are sensitive to the increase in skills and knowledge resulting from such a course but not to learners' levels. Validity evidence from scores for the CPT confirms implementation in new contexts and partially supports internal structure. More evidence is required to further support internal structure and especially to support relations with other variables and consequence evidence. Additional modifications should be made to the CPT before considering its use for high-stakes certification such as PALS.
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.120 | 0.163 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.001 |
| Bibliometrics | 0.002 | 0.002 |
| Science and technology studies | 0.001 | 0.003 |
| Scholarly communication | 0.002 | 0.001 |
| Open science | 0.002 | 0.003 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.002 | 0.001 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".